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SANTRAUKA 

Atliekant bet kokius jvertinimus, visada at- 
siranda interesg konfliktu tarp socialiniij 
dalininkij ir jiems patiems. Norint paaiskinti 
tokias antinomijas, buvo sukurta taksonomi- 
ja, pristatoma ir aptariama siame straipsnyje. 
Kaip pavyzdys panaudotas universiteti} jver- 
tinimo procesas, atliekamas nepriklausomij 
ekspertg. 

Taksonomijayra keturiij dimensijp: a) is- 
skiriamos septynios vertybip sritys (etines, 
metodines, socialines ir interaktyvios, teisi- 
nes, ekonomines, asmenines, visuomenines 
ir politines); b) antinomijos gali buti „arba... 
arba" (dichotomija; „alternatyvi antinomi- 
ja“) arba „daugiau ar maziau" tipo (labiau 
isreiskiamas kazkuris klausimas; „palaips- 
nine antinomija"); c) egzistuoja tikslij svar- 
bos antinomijos (priestaringi to paties arba 
kito socialinio dalininko tikslai, t. y. skirtin- 
gi tikslai negali buti pasiekti vienu metu) ir 
priemoniij masto antinomijos (nors tikslai 
ir nepriestarauja vieni kitiems, jp pasiekimo 
priemones nesuderinamos), abiem atvejais 

1 J.-L. Patry dalj straipsnio parase budamas Melburno Mo¬ 
nash universiteto Australijoje kviestiniu mokslininku. 

2 Autoriai dekoja dviem anonimiskiems ekspertams ir 
Denis Phillips bei K^stuciui Pukeliui uz naudingas pastabas 
ankstesnems sio rankrascio versijoms. 


ABSTRACT 

Whenever evaluations are done, there are an¬ 
tinomies of interests both within and between 
stakeholders. To account for such antinomies, 
taxonomy has been developed which will 
be presented and discussed using the peer- 
review processes in university evaluations as 
example. 

The taxonomy contains four dimensions: 

a) seven values domains are distinguished 
(ethical, methods, social and interactive, legal, 
economic, personal, and societal values); an¬ 
tinomies can be within or between domains; 

b) antinomies can be of the either-or-type 
(dichotomy; “alternative antinomy”), or they 
can be of the more-or-less type (more of one 
issue implies less of the other; “gradual an¬ 
tinomy”); c) there are antinomies on the goal 
dimension (different goals of the same or of 
different stakeholders are in antinomy, i.e., 
cannot be achieved simultaneously) and on 
the means dimension (although the goals are 
not in antinomy, the means to achieve them 


1 Part of Patry s contribution was written while he was 
a visiting scholar at the Monash University, Melbourne, 
Australia. 

2 The authors wish to thank two anonymous reviewers and 
Denis Phillips and Kestutis Pukelis for their helpful remarks 
to earlier versions of this manuscript. 
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gali buti taikoma alternatyvi ir palaipsnine 
antinomija; d) antinomijos atveju siuloma ir 
jrodinejama normatyvine sprenditmj priemi- 
mo hierarchy a. 

Nepriklausomiems ekspertams verti- 
nant aukstojo mokslo jstaigas, galima ap- 
tikti jvairius antinomy ij tipus. Pateiksime 
praktinio vertinimo metu aptiktus pavyz- 
dzius ir detaliai isanalizuosime prototipin^ 
antinomy y: nepriklausomi ekspertai laiko- 
mi socialiniais dalininkais, todel bus aptarti 
jy interesai ir tikslai (vieno socialinio dali- 
ninko vidaus antinomijos). Taikant norma- 
tyvin^ hierarchija, galima atrasti sprendimy, 
bet negalima pakeisti vertintojy asmeniniu 
sprendimij. 

PAGRINDINigTERMINg APIBREZIMAI 

• Aukstojo mokslo institucijos - tai vertini¬ 
mo antinomija analizes duomenij baze, kuriy 
nepriklausomi ekspertai, dirbantys jvairiose 
aukstojo mokslo institucijose - universitetij 
katedrose, taikomuosiuose universitetuose, 
pedagoginiuose universitetuose ir pan. (Aus- 
trijoje, Sveicarijoje, Lietuvoje ir kitose salyse), 
vertina remdamiesi savo patirtimi. 

• Ekspertinis vertinimas - eksperty koman- 
dos atliekamas vertinimas apie institucijos 
veiksmingumy specifiniais aspektais, pavyz- 
dziui svietimo, tyrimij ir administravimo 
klausimais ir pan. Dalis eksperty yra to paties 
akademinio lygmens kaip ir vertinami kole- 
gos bei dirba panasiose institucijose. 

• Vertybii^ dilemos - kai veiksmas sutelk- 
tas j tam tikras vertybes, pavyzdziui, etines, 
asmenines, ekonomines ar teisines, t. y. „di- 
lemos“ atveju protagonistas gali atlikti P 
arba P , bet ne abu. Kitij antinomijij atveju 
nesuderinamumas nera grieztas - labiau sie- 
kiama kompromiso tarp dvieji} vertybii}: ry- 
siai tarp P ir P 2 priklauso tipui daugiau P , 
maziau P 2 ; mes tai vadiname palaipsninemis 
antinomijomis. 


are incompatible), on both dimensions there 
can be alternative and gradual antinomies; d) 
a normative hierarchy for decision making in 
the case of antinomies is proposed and argued 
for. 

In evaluating higher education institu¬ 
tions with peer review, all types of antinomies 
can be found. Examples for such antinomies 
we have encountered in practical evaluations 
will be given, and a prototypical antinomy will 
be analyzed in detail: The peers are seen as 
stakeholders, and their interests and goals will 
be discussed (antinomies within one stake¬ 
holder). Applying the normative hierarchy can 
help to find a solution, yet it cannot replace the 
evaluators’ personal decision. 

DEFINITIONS OF KEYTERMS 

• Higher education institutions - the database 
for the analysis of antinomies in peer review 
valuations of higher education institutions is 
the authors’ own experience as peer review¬ 
ers in higher education institutions of different 
kinds: University departments, Applied Uni¬ 
versities, Teachers’ Colleges, etc. in Austria, 
Switzerland, Lithuania and other countries. 

• Peer Review - the aim of a peer reviewing 
evaluation is typically to check the institu¬ 
tion’s effectiveness by a team of experts with 
respect to specific aspects, such as education, 
research, and administration. At least some of 
the reviewers are on the same academic level 
(“peers”) as the evaluees and work in similar 
institutions. 

• Values dilemmas - in a “dilemma” the pro¬ 
tagonist can do either P or P„ but no combi¬ 
nation of both. The action is focused on certain 
values, like ethical, personal, economic, and 
legal values. In other antinomies the mutual 
exclusion is not strict, instead there is a trade¬ 
off between the two values: the relationship 
between P and P 2 is of the type the more P , 
the less P 7 ; we call it gradual antinomies. 
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ANTINOMIJOS SOCIAUNIUOSE 
TINKLUOSE 

Socialiniij tinklij susikirtimo taskuose gali 
susidaryti antinomijos tarp susijusiij asmenij 
ir institucijij, nes skirtingi zmones gina skir- 
tingus ir daznai priestaringus tikslus bei in- 
teresus. Tai tinka ir ivertinimij atzvilgiu: bet 
kuris socialinis dalininkas (ivertintojus pri- 
skiriame prie socialiniij dalininkij) susiduria 
su keliomis antinomijomis tarp vertybiij. 

|vertinimo procesij metu tokios antino¬ 
mijos tarp vertybiij pasitaiko gana daznai: 
dauguma ivertintojij (Turner, 2003) arba 
bent jau du trecdaliai (Morris, Cohn, 1993) 
teige susidur^ su etinemis antinomijomis. 
Trecdalis fondij direktoriij, finansavusiij 
jvertinimus, pranese apie etines antinomijas 
(Morris, 2007). Taigi tokios antinomijos yra 
ne isimtis, o taisykle. )ij, kaip laiko momen¬ 
ta ar pavienes spr^stinos problemos, mode- 
liavimas nera pakankamas (zr. Meter, 2006, 
151). jvertinimas turi buti labiau laikomas 
procesu su nuolat antinomiskais tikslais ir 
interesais. PasakM. Morris (2007, 413), „at- 
rodo, kad daug sios srities sunkunuj vis dar 
ignoruojama, del ko veliau vertinant atsiran- 
da antinomijij“. Jas nagrineti - isankstine sij- 
lyga ir pirmasis zingsnis, jei ne j sprendimo, 
tai bent j supratimq ir racionalij tvarkymqsi, 
kas gali reiksti apgalvotij ir pagrjstq pasirin- 
kim;j is keliij variantij. Siame darbe pirmiau- 
siai pristatysime pagrindinius duomenis apie 
naudojamq ivertinimij tipij: aukstojo mokslo 
institucijij nepriklausomij ekspertij jvertini- 
nnj. Antra, pateiksime hierarchinq taksono- 
mijq antinomijij jvertinimams, kuriq sudaro 
keturios tarpusavyje susijusios dimensijos 
(vertybiij sritys; alternatyvi ir palaipsnine 
antinomijos; antinomijos tikslo ir priemo- 
nip lygmenimis; siuloma principij, taikomij, 
susidurus su antinomija, hierarchija). Trecia, 
panaudosime taksonomijij prototipinems 
antinomijoms analizuoti nepriklausomij eks- 
pertij atliktuose aukstojo mokslo institucijij 


ANTINOMIES IN SOCIAL NETWORKS 

In the intersection of social networks an¬ 
tinomies between the involved persons and 
institutions may occur since different people 
defend different and often contradicting aims 
and interests. This holds also in evaluations: 
any stakeholder (and we include the evalua¬ 
tors among the stakeholders) is confronted 
with several antinomies between values. 

In evaluation processes such antinomies 
between values are quite frequent: most eval¬ 
uators (Turner, 2003) or at least two third of 
them (Morris & Cohn, 1993) reported that 
they had encountered ethical antinomies. As 
to the directors of foundations that financed 
evaluations, one third reported ethical antino¬ 
mies (Morris, 2007). Hence such antinomies 
are not the exception, but the rule. Modeling 
them as a moment in time, or as a single prob¬ 
lem to be resolved, will not suffice (see Meter, 
2006, 151), rather evaluation must be seen as 
process with persistently antinomic aims and 
interests. According to Morris (2007, 413), “it 
appears that many challenges in this domain 
continue to slip under the radar, causing an¬ 
tinomies later in the evaluation”. Addressing 
them is a precondition and the first step if 
not towards a solution, so in any case towards 
awareness and dealing rationally with them, 
which might mean a deliberate, yet justified 
choice among the options. In this paper we 
present, first, a short background of the type 
of evaluations with which we are dealing: peer 
review evaluation of higher education insti¬ 
tutions. Secondly we introduce a hierarchi¬ 
cal taxonomy of antinomies in evaluations, 
consisting in four interrelated dimensions 
(domains of values; alternative and gradual 
antinomies; antinomies on the goal and on 
the means levels; and a proposed hierarchy of 
principles to be applied when confronted with 
an antinomy). Thirdly we use the taxonomy to 
analyze prototypical antinomies in peer review 
evaluations of higher education institutions. 
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jvertinimuose. Galiausiai aptarsime rysius 
tarp skirtingij taksonomijos dimensijq ir jij 
naudingumq. 

1 

AUKSTOJO MOKSLO INSTITUCIjg 
EKSPERTINIS VERTINIMAS 

Nepriklausomij ekspertij atliekamas vertini- 
mas - vienas is jvertinimo tipij. Jj organizuoja 
jgaliojanti jstaiga (nepriklausoma jstaiga ar 
politinis priziuretojas) arba pati instituci¬ 
ja. Pastaruoju atveju institucija veikia ir kaip 
jgaliojanti jstaiga, bet dazniausiai jvertinimas 
inicijuojamas del isorinio spaudimo. Paprastai 
tikslas buna patikrinti institucijos veiksmin- 
gum;j specifiniais aspektais, pavyzdziui, del 
svietimo, tyriimj ir administravimo. Demesys 
buna sutelktas j: 1) institucijos darbq , kurj rodo 
joje dirbanciij ar besimokanciij asmenij arba 
asmenig kaip nors susijusiij su svietimo ins¬ 
titucijos darbo ir mokymosi procesais (tai yra 
kai kurie is vertinimo socialiniij dalininkij), 
pasiekimai, poziuris ir pan.; 2 )faktus ir turinj, 
pavyzdziui, studijij turinj, mokslo program^, 
mokslinius pajegumus ir pan. Jij autoriai gali 
buti kita socialiniij dalininkij grupe, 3) proce- 
sai ir pagrindines strukturos (pvz., sprendirmj 
priemimo strukturos ir procesai, organizavi- 
mas ir pan.), skirti siam procesui atlikti arba 
sukurti tokius faktus ir turinj. 

Paprastai procedura vyksta taip: pasi- 
renkama skirtingas kvalifikacijas turinciij 
ekspertij komanda (maziausiai vienas eks- 
pertas pagal tem;j, kuria rupinasi institucija, 
destymo ir studijavimo ekspertas, vertinimo 
ekspertas; daznai buna ir administravimo 
ekspertas, prireikus - ir praktinio pritaikymo 
ekspertas). Kai kurie jij yra to paties akademi- 
nio lygmens („kolegos“), kaip vertinamieji, ir 
dirba panasiose institucijose - sis darbas su¬ 
telktas butent j tokius jvertintojus. Paprastai 
ekspertai privalo deklaruoti nepriklausantys 
institucijai, kuri bus jvertinama. 


Finally the relationships between the different 
dimensions of the taxonomy and their useful¬ 
ness are discussed. 

1 

PEER REVIEW EVALUATIONS OF 
HIGHER EDUCATION INSTITUTIONS 

A peer reviewing evaluation is one type of eval¬ 
uation among others. It is organized either by 
a mandating agency (an independent agency 
or the political supervisor) or the institution 
itself; in the latter case the institution is also the 
mandating agency, but initiates the evaluation 
most often under some external pressure. The 
aim is typically to check the institutions effec¬ 
tiveness with respect to specific aspects, such as 
education, research, and administration. The 
focus is (1) on the performances of the institu¬ 
tion, represented through the achievements, at¬ 
titudes, etc., of persons working or studying at 
that institution or who are involved in any way 
in the working and learning processes of the 
educational institution (these are some of the 
stakeholders of the evaluation); (2) on the facts 
and contents such as the curriculum, the scien¬ 
tific program, the scientific output, etc., whose 
authors may be another group of stakeholders, 
(3) on the processes and underlying structures 
(e.g., decision making structures and process¬ 
es, the organization, etc.) to reach those perfor¬ 
mances or to develop these facts and contents. 

The procedure is typically that a team of 
reviewers of different qualifications (usually 
at least an expert on the topic the institution 
is concerned with, an expert on teaching and 
learning, and an expert on evaluation; often 
also an expert in administration and if ap¬ 
propriate one dealing with practical applica¬ 
tion) is chosen. At least some of the reviewers 
are on the same academic level (“peers”) as the 
evaluees and work in similar institutions - the 
focus of the present paper lies on these evalu¬ 
ators. The reviewers have usually to declare 
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Pasak F. A. van Vught ir D. F. Westerheij- 
den (1994), vertinant aukstqjj moksla, budin- 
gi keturi elemental: vadovaujantis atstovas, 
jsivertinimas, ekspertinis vertinimas apsilan- 
kius vietoje ir ataskaita. Tiksliau, vertinimo 
procedures, kuriose dalyvavome, sudare to- 
kie zingsniai (zr. Speer, 2010; panasaus sqraso 
1 langelis): 

1. jgaliojanti jstaiga (veikianti ir kaip 
jgalioj antis atstovas) sudaro vertinimo dar- 
botvarkq, j kuriq jtraukiami jvertinimo metu 
atsakytini klausimai. 

2. Vertintojus (kolegas ir kitus) parenka 
jgaliojanti jstaiga. 

3. Jvertinta institucija atlieka jsivertini- 
mq, kurio rezultatai pateikiami ataskaita. 

4. Ekspertai perskaito ataskaitq, papraso 
papildomos medziagos, jeigu reikia, ir sudaro 
klausimij, j kuriuos tures atsakyti institucija, 
sqrasq. 

5. Institucija suteikia reikalingq pa- 
pildomq medziagq ir atsako j kai kuriuos 
klausimus. 

6. Ekspertai apsilanko institucijoje ir 
bendraudami su jos zmonemis bei budami 
vietoje iesko atsakymij j klausimus; daznai tai 
apima pirmqjq atsakomqjq reakcijq. 

7. Ekspertai paraso ataskaitq. 

8. Paprastai institucijai suteikiama gali- 
mybe pateikti pastabas del ataskaitos. 

9. Vertintojai (ekspertai) sudaro galutiny 
ataskaitq ir jteikia jq jgalioj anciai jstaigai. 

10. jgaliojanti jstaiga imasi tolesniij zings- 
niq (jvertinimo pasekmes ir pan.). 

Procedures gali skirtis, bet bendra struk- 
tura paprastai buna gana panasi. Si procedure 
ir taikoma antinomijij analizei; naudojant ki- 
tokicj procedure, reikia atitinkamai pritaikyti 
analiz^. 

Nepriklausomp eksperty aukstojo 
mokslo instituciji} jvertinimo antinomiji} 
analizes duomenp baze - tai autorip, kaip 
nepriklausomij ekspertp, patirtis skirtingose 
aukstojo mokslo institucijose: universitetp 
katedrose, taikomuosiuose universitetuose, 


their independence from the institution to be 
evaluated. 

According to van Vught and Westerhei- 
jden (1994) four elements are characteristic for 
evaluations in higher education: a managing 
agent, a self-evaluation, a peer review with site 
visit, and a report. More precisely, the evalua¬ 
tion procedures we were involved in consisted 
in the following steps (see also Speer, 2010, 
box 1, for a similar list): 

1. The mandating agency (which acts also 
as managing agent) formulates an evaluation 
agenda including the questions to be answered 
within the evaluation. 

2. The evaluators (peers and others) are 
chosen by the mandating agency. 

3. The evaluated institution performs 
a self-evaluation whose results are document¬ 
ed in a report. 

4. The reviewers read the report, ask for 
additional material if necessary, and formu¬ 
late a question pool to be answered by the 
institution. 

5. The institution provides the required 
additional material and responds to some of 
the questions. 

6. The reviewers visit the institution and 
through interaction with the people of the institu¬ 
tion and a site visit look for answers to the ques¬ 
tions; frequently this includes a first feedback. 

7. The reviewers write a report. 

8. Usually the institution has the opportu¬ 
nity to comment on the report. 

9. A final report is authored by the peers 
and handed over to the mandating institution. 

10. The mandating institution takes the fur¬ 
ther steps (consequences of the evaluation, etc.). 

The procedures may vary, but the general 
structure is usually quite similar. This will be 
the procedure assumed for the analysis of the 
antinomies; if a differing procedure is used, 
the analysis needs to be adapted. 

The database for the analysis of antinomies 
in peer review evaluations of higher educa¬ 
tion institutions to be presented below is the 
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pedagoginiuose universitetuose ir pan. 
Austrijoje, Sveicarijoje, Lietuvoje ir kitose 
salyse; del etiniq priezasciq sip instituci- 
jp nenurodome. jvertinimo komitetuo- 
se straipsnio autoriai paprastai atlikdavo 
destymo ir studijavimo ekspertp funkci¬ 
jas, paprastai programos didaktines dalies 
jvertinimp; pedagoginiuose universitetuose 
visi kolegos buvo destymo ir studijavimo, 
bet skirtingp sricip specialistai: straipsnio 
autoriai - svietimo tyrejai, dauguma kitp 
kolegp - universitetp, kurip prioritetai ne 
tokie moksliski, destytojai. Be to, straipsnio 
autoriai daznai atliko jvertinimo specialistp 
funkcijas jvertinimo grupeje; butent atlik- 
dami sias funkcijas dazniausiai susidurdavo 
su antinomijomis. 


2 

KETURiy DIMENSIJy ANTINOMIjy 
TAKSONOMIJA VERTINANT 

Antinomija - tai situacija, kai protagonistas 
gali pasirinkti vienp is dviejp galimybip - 
P arba P Kiekviena jp pasizymi tam tikra 
verte (arba vercip rinkiniu), bet pasirinkus 
P galimyb^, nebus pasiektos vertybes, sie- 
jamos su galimybe P 2 , o pasirinkus P 2 , su P t 
susijusios vertybes tampa nejmanomos. Kad 
ir kp protagonistas darytp, vertybes, sieja- 
mos su viena ar kita galimybe, bus atmestos. 
Galima sakyti, kad vertybes, susijusios su P 
ir P 2 yra antagonistines. Vertybes, kuriomis 
rizikuojama siose antinomijose, nepriklauso 
vertintinp programp ar institucijp verty- 
bems, nebent priestarautp vertintojp gina- 
moms ar kitoms jvertinimo procesui svar- 
bioms vertybems. 

Daznai egzistuoja daugiau nei dvi ga- 
limybes ir isplaukiancios vertybes. Pavyz- 
dziui, S. Delamontas (2005) nurodo keturis 
antagonistinius vertybip rinkinius, is kurip 


authors’ own experience as peer reviewers in 
higher education institutions of different kinds 
(University departments, Applied Universities, 
Teachers’ Colleges, etc.) in Austria, Switzer¬ 
land, Lithuania and other countries; for ethical 
reasons we do not disclose the respective insti¬ 
tutions. The authors’ function in the evaluation 
committees was usually the expert in teaching 
and learning focusing particularly on the eval¬ 
uation of the education part of the program; in 
Teachers’ Colleges, all peers were specialists of 
teaching and learning, but with different foci: 
the authors were the researchers in education; 
mostly the other peers where teachers from 
other Colleges whose priorities were less sci¬ 
entific. In addition, the authors had often the 
function of evaluation specialists within the 
evaluation group; it is in this function that they 
encountered antinomies most frequently. 

2 

A FOUR-DIMENSIONAL TAXONOMY 
OF ANTINOMIES IN EVALUATION 

An antinomy is a situation in which the pro¬ 
tagonist has the choice between two possibili¬ 
ties, P and P . Each of these possibilities has a 
certain value (or set of values) for him or her, 
but choosing possibility P will result in not 
achieving the values associated with possibil¬ 
ity P , and doing P 2 renders the values related 
with P impossible. Whatever the protagonist 
does will break the value related with one or 
the other possibility. One can also say that the 
values related with P and P 2 are antagonistic. 
The values that are at stake in these antino¬ 
mies are not the values of the programs or in¬ 
stitutions to be evaluated unless these are in 
opposition to the values the evaluators defend 
or other values that are important within the 
evaluation process. 

Often there are more than two possibili¬ 
ties and ensuing values. Delamont (2005) for 
instance mentions four antagonistic sets of 
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jvertintojas gali rinktis: praktin^ vertyb^, 
mokslinj reiksmingumq ( immortality ), zlug- 
ti pasmerktq projektq ir „nezinomq pasauliq 
stebuklq tyrinejimq". Del paprastumo nagri- 
nesime dvi vienalaikes vertybes. 

Jeigu nenurodyta kitaip, siame darbe 
sqvoka vertybes aprepia daugiau nei jpras- 
tai literaturoje vartojama sqvoka. Ji apima 
tai, kjj. atitinkamas asmuo laiko svarbiais 
dalykais, pvz., jos ar jo interesai, susij^ su 
tuo kq butina pasiekti ir pan.; tam tikru 
poziuriu vertybes gali buti laikomos tiks- 
lais, kuriuos kazkas nori jgyvendinti. Siame 
tekste terminas tikslas priesinamas priemo- 
nems, naudojamoms tiems tikslams pasiek¬ 
ti (zr. 3 taksonomijos dimensijq), ir siekiant 
isvengti painiavos nebus vartojamas kita 
reiksme. 

Be to, reikia pamineti, kad dauguma ver- 
tybiij formuojamos kaip maksimumas (pvz., 
„gauti kuo daugiau informacijos“). Straipsnio 
autoriai puikiai supranta, kad nedera susi- 
telkti tik ties siomis vertybemis ir kad daugu¬ 
ma tokiij maksimuim} yra nepasiekiama. Jlj 
netinkamumo ir nepasiekiamumo priezastis 
ta, kad jos visada antinomiskos kitoms verty- 
bems; norint nagrineti sias antinomijas, reikia 
tiesiai nurodyti vertybes, kad butij atskleistas 
jq vaidmuo antinomijose ir atitinkamos prie- 
singos vertybes. 

Taksonomija, kuriq pateiksime, yra ketu- 
riij tarpusavyje susijusiij dimensijq: 

• Pirmiausia isskiriamos septynios sri- 
tys, svarbios jvertinimo tyrimams; 

• Antra, isskiriami du antinomijij tipai: 
alternatyvios antinomijos (dilemos) ir pa- 
laipsnines antinomijos; 

• Trecia, isskiriamos tikslp svarbos ir 
priemoniij masto antinomijos; 

• Ketvirta, siuloma bandomoji principij, 
kurie bus naudojami antinomijas jvertinimo 
tyrimij metu, hierarchija. 

J.-L. Patry ir A. Gastager (2004) pritaike 
pirmas tris dimensijas mokymui jvertinti tam 
tikrame universitete. Siame darbe pirmoji 


values an evaluator may have to choose from: 
practical value, scholarly immortality, project 
doomed to fail, and explore the wonders of 
unknown worlds. For sake of simplicity, we 
will deal only with two simultaneous values. 

Unless specified otherwise, throughout 
this paper the concept of values is broader than 
typically used in the literature; it includes what 
is regarded as important, relevant or necessary 
to achieve from the standpoint of the respective 
person and includes his or her interests, etc. In 
some way, values can be seen as goals someone 
wants to achieve; however, in this text the term 
“goal” is opposed to the means that someone 
uses to achieve these goals (see the third di¬ 
mension of the taxonomy below) and will not 
be used elsewhere to avoid confusion. 

It must also be mentioned that many of the 
values are formulated as maximum (e.g., “get¬ 
ting as much information as possible”). The 
authors are well aware that focusing uniquely 
on such values is inappropriate, and that many 
of these maximum values are unattainable. 
The reason for their inappropriateness and un¬ 
attainability is always that they are antinomic 
with other values; to address these antinomies 
it is necessary to put the values bluntly so that 
their role in the antinomies and the respective 
opposite values can be revealed. 

The taxonomy to be presented has four in¬ 
terrelated dimensions: 

• First, seven domains that are relevant in 
evaluation studies are distinguished. 

• Second, two types of antinomies are dis¬ 
tinguished: alternative antinomies (dilemmas) 
and gradual antinomies. 

• Third, antinomies on the goal and on 
the means dimensions are distinguished. 

• Fourth, a tentative hierarchy of princi¬ 
ples to be used when trying to solve antino¬ 
mies in evaluation studies is proposed. 

The first three dimensions have been ap¬ 
plied to the evaluation of teaching in a spe¬ 
cific university in Patry and Gastager (2004). 
In the present paper, the first dimension has 
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dimensija siek tiek patobulinta, prideta ket- 
virtoji dimensija ir visos pritaikytos nepri- 
klausomam aukstojo mokslo institucijp eks- 
pertiniam jvertinimui. 

2.1. VERTYBig SRITYS 

Norint identifikuoti vertybiij antinomijas 
vertinant, pirmiausia reikia isskirti tas, ku- 
rios yra „statomas ant kortos“ (svarbiausios). 
Vengiant siauro poziurio, naudinga sudaryti 
galimij antinomijoje busiancip vertybip sp- 
rasp, kad nebutp pamirsta ne viena svarbi 
sritis. 

Vertintojij (Gastager, Patry, 2006) antino- 
mijp meta-jvertinimo metu buvo isskirtos se- 
sios vertybii} sritys: etines, metodines, socia- 
lines ir interaktyviosios, teisines, ekonomines 
ir asmenines; velesni tyrimai parode, kad 
reikalinga ir septinta sritis - visuomenines 
vertybes. Sesios is septynip sriciu minimos 
literaturoje apie jvertinimp; retai kalbama 
tik apie ekonomin^ sritj. Siame straipsnyje 
aptariant sritis pateikiami keli pavyzdziai is 
ekspertinio aukstojo mokslo institucijij verti- 
nimo. Pavyzdziai nera issamus, tik iliustruoja 
svarbiausias vertybes. 

1. Etines vertybes apima etiskai pagrjstas 
vertybes: spziningump ir skaidrump (pvz., 
aiskinant vertinimo priemones ir pagrindi- 
nius dalykus tiriamiesiems). Kita vertybe - 
pazado laikymasis (pvz., del konfidencialu- 
mo). Tokio pobudzio vertybes aptariamos 
per etinius debatus, tiriant jvertinimp (pvz., 
Morris, 2005; Schwandt, 1997; Simons, 2006); 
dauguma vertinimo asociacijij yra sukuru- 
sios etines normas (pvz., Amerikos jvertini- 
mo asociacija, 2004; Australazijos jvertinimo 
bendrija, 2000; Jungtinis svietimo jvertinimo 
standartij komitetas, 1994), o visi vertintojai 
vienaip ar kitaip atsizvelgia j etinius aspek- 
tus. Pavyzdziui, W. Shadish, T. D. Cook ir 
L. C. Levington (1991) atskleide ivairiij auto- 
rip etines nuostatas vertinant; vienos nuoro- 
dos j etikp yra issamesnes uz kitas, skiriasi ir 
autorip poziuriai. 


been improved slightly, the fourth dimension 
has been added, and the application is to peer 
reviewing in the evaluation of higher educa¬ 
tion institutions. 

2.1. DOMAINS OF VALUES 

To identify values antinomies in evaluation, it is 
first necessary to distinguish the values that are 
at stake in an evaluation. In order to avoid being 
too narrow in doing this it is useful to have a hst 
of potential values that could be in antinomy so 
that no important domain gets forgotten. 

In a meta-evaluation study of evaluators’ 
antinomies (Gastager & Patry, 2006) six areas 
of values were distinguished: Ethical, methodi¬ 
cal, social and interactive, legal, economic, and 
personal values; further analyses showed that a 
seventh domain was necessary: societal issues. 
Six of seven domains can be found in the litera¬ 
ture on evaluation; only the economic domain is 
rarely addressed. The domains are presented be¬ 
low with a few examples from peer review evalu¬ 
ations of higher education institutions; these ex¬ 
amples are by no means exhaustive, but rather 
serve as illustrations for the values at stake. 

1. Ethical values include values based on 
ethical justification. Examples for such values 
are honesty and transparency (e.g., explain¬ 
ing the instruments and the central issues of 
the evaluation to the investigated persons). 
A further value is holding ones promise (for 
instance regarding confidentiality). Values of 
this kind are discussed in the ethical debates 
in evaluation research (e.g., Morris, 2005; 
Schwandt, 1997; Simons, 2006), most evalua¬ 
tion association have developed ethical guide¬ 
lines (e.g., American Evaluation Association, 
2004; Australasian Evaluation Society, 2000; 
Joint Committee on Standards for Educational 
Evaluations, 1994), and all evaluators include, 
in one way or another, ethical reflections. 
Shadish, Cook, and Levington (1991), for in¬ 
stance, showed the ethical backgrounds of dif¬ 
ferent important authors in evaluation; some 
references to ethics are more explicit, others 
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Reikia pamineti, kad cia vartojama etiniij 
vertybiij s^voka yra siauresne uz bendnjjj. 
Etines vertybes aptariamos per etinius de- 
batus, pagrindinj demesj skiriant tam, kaip 
zmones turetij elgtis vieni su kitais; siuo kon- 
kreciu atveju - kaip jvertintojai turetij elgtis 
su dalyvaujanciais asmenimis (socialiniais 
dalininkais, zmonemis, dirbanciais aukstojo 
mokslo institucijoje ir pan.). 

Tipiskos nepriklausomi} ekspertij eti¬ 
nes vertybes - etiskas elgesys, pavyzdziui, 
gerbiamas dalyvio savarankiskumas, infor- 
macija del sutikimas, vengiama skriaudos ir 
apgaules, ir pan.; s^ziningumas ir skaidru- 
mas, pavyzdziui, vertybiij detalizavimas, su- 
sikoncentravimas j faktus, o ne j insinuacijas, 
ir pan.; konfidencialumas, vadinasi, negalima 
atskleisti informantij vardij (anonimiskumas 
nejmanomas, nes skirtingij grupiij atstovai 
susitinka asmeniskai); taip pat gerbtinas ins- 
titucijos poreikis gauti griztam;y;j informaci- 
ja ir reaguoti j j;j. 

2. Metodais renkama informacija - tai 
pavyzdziij rinkimas, piano sudarymas, ver- 
tinimas, tiketinij konkuruojanciij hipote- 
ziij atmetimas, tinkama duomenij analize 
ir interpretavimas, ir pan. Tam priklauso 
ir metateorines, metaetines bei metodolo- 
gines vertybes, pabreziancios pagrindinius 
vertinimui naudojamus metodus. Ko gero, 
metodai yra labiausiai aptarinejami vertini- 
mo klausimai; M. C. Alkin ir C. A. Christie 
(2004, p. 13), D. T. Campbell, T. D. Cook ir 
L. J. Cronbach suvokia vienij is trijp „jverti- 
nimo medzio“ sakij kaip metodij sak^; kita 
vertus, E. Guba ir Y. S. Lincoln (1989, 2000), 
aprasydami medzio „ivertinim;j“, kritikuoja 
siuos metodus. 

Bendras metodij vertybiij tikslas - gauti 
optimalius duomenis apie institucikj; siame 
kontekste sijvokij optimalus autoriai gali su- 
prasti skirtingai. Vis delto, metodai nagrineja 
klausimij, kaip vertintojai renka duomenis ir 
argumentuoja savo teiginius (jskaitant ir savo 
padarytas isvadas). Nors metodais kartais 


are rather implicit, and there are many differ¬ 
ences between authors. 

It must be mentioned that the concept of 
ethical values used here is narrower than the 
concept of values in general (see above). Ethi¬ 
cal values are values which are addressed in 
ethical debates, the focus being on how people 
should deal with each other, in this particular 
case how the evaluators should deal with the 
people involved in the evaluation (stakehold¬ 
ers, people working in the higher education 
institution, etc.). 

Typical ethical values of peer reviewers 
are ethical conduct like respecting the partici¬ 
pants’ autonomy, informed consent, avoiding 
harm and deception, etc.; honesty and trans¬ 
parency, like making values explicit, focusing 
on facts instead of insinuations, etc.; confiden¬ 
tiality, which means not revealing the names 
of the informants (anonymity is not possible 
since in the site visit the representatives of the 
different groups of the institution are met in 
person); and respecting the institution’s need 
for information through feedback and giving 
the possibility to respond. 

2. The methods include everything that is 
done to gather valid information, such as sam¬ 
pling, design, assessment, ruling out plausible 
rival hypotheses, but also appropriate analysis 
and interpretation of the data, etc. Included 
are also the meta-theoretical, meta-ethical, 
and methodological values underlying the ac¬ 
tual methods used in the evaluation. Methods 
belong probably to the most discussed issues 
in evaluation; Alkin and Christie (2004, p. 13) 
conceive one of the three main branches of their 
“evaluation tree” as the methods branch with 
authors like Campbell, Cook, and Cronbach; on 
the other hand, authors like Guba and Lincoln 
(1989,2000), who are depicted in the “valuing” 
branch of the tree, criticize these methods. 

Methods values serve the general aim of 
gaining optimal information about the institu¬ 
tion; different authors may differ in what they 
consider optimal in this context. In any case, 
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suabejojama (pvz., rezultatais nepatenkinti 
socialiniai dalininkai; zr. Chelimsky, 2006), 
susijusius klausimus paprastai sprendzia jver- 
tinimo specialistai. 

Visais ivertinimq atvejais viena svar- 
biausiq vertybiq - pagrjstumas (validity). 
Tai suprantama kaip idealus niekieno ne- 
paveiktas vertinimas; kadangi tai nerealu, 
svarbu pagrjsta teorija paaiskinti saliskumq 
(pvz., Patry, 2011). Visos pastabos priklauso 
nuo teorijos (Hanson, 1958), tad vertinto- 
jams zinoti pagrindin^ teorijq, net jeigu ver¬ 
tinimas ir nera skirtas teorijoms patikrinti. 
Del epistemologijos skirtumij (zr. Alkin, 
Christie, 2004; Shadish ir kt., 1991; ir kt.) 
reiketq atsizvelgti j asmens nuomon^ - juk 
jvertinimo komiteto ekspertij poziuriai gali 
skirtis, o neissiaiskinus kyla nesusipratimai 
ir painiava. 

3. Socialities ir interaktyvios vertybes su- 
sijusios su elgesiu ir aprepia pagarbq, man- 
dagumq, bendravimo stiliq, derybas ir pan., 
jautrumq (Forss ir kt., 2002, p. 35), taktq (van 
Manen, 1991) arba tinkamus rysius (Abma, 
2006) bendraujant ir pan. Sqvokos pataria- 
masis demokratinis poziuris (House, 1980; 
House, Howe, 2000; ir pan.), ketvirtos kar- 
tos jvertinimas (Guba, Lincoln, 1989) ar su- 
teikiantis mokejimif jvertinimas (Fetterman 
ir kt., 1996) dazniausiai vartojamos siems 
tikslams. 

Sios srities vertybes - socialines psicho- 
logijos tema, kur pagrindinis demesys skiria- 
mas socialiniams rysiams optimizuoti, o viskp 
siejanti vertybe - renkant butin;j informacij;j 
gerbti partnerip orum^, asmenyb^, poreikius 
ir savarankiskum^. 

Bet koks socialinis bendravimas ir tinka- 
mas elgesys priklauso nuo situacijos (Patry, 
2011), vadinasi, vertintojai turi atsizvelgti j 
kontekstp Kitaip sakant, vertinant jp vaid- 
muo yra kitoks nei jprastai bendraujant, susi- 
tikus kitomis aplinkybemis, - tokiose mazose 
salyse kaip Austrija, Sveicarija ar Lietuva tai 
atsitinka daznai. 


methods address the question how the evalu¬ 
ators proceed in gathering data and in arguing 
their statements (including the conclusions they 
reach). Although sometimes methods are being 
challenged from the outside (e.g., by stakeholders 
who are not satisfied with the results, see Chelim¬ 
sky, 2006, for an example), the relevant issues are 
typically dealt with by specialists in evaluation. 

In all evaluations one of the most impor¬ 
tant values is validity. This includes the ideal 
of uninfluenced assessment; because this is not 
realistic, it is important to account for biases 
with a reasonable theory (see, for instance, Pa¬ 
try, 2011). Further, since all observation is the¬ 
ory-dependent (Hanson, 1958), it is necessary 
that the evaluators are aware of their underly¬ 
ing theory even if evaluation does not aim at 
testing theories, and because of the differences 
in epistemology (see Alkin & Christie, 2004; 
Shadish et al„ 1991; and others), it is also ap¬ 
propriate to be aware of one’s stand in this re¬ 
gard - it might well be that the peers within 
an evaluation committee might have different 
opinions which, if not made explicit, may lead 
to misunderstandings and confusion. 

3. Social and interactive values concern 
requests and expectancies about dealing with 
each other and include courtesy, politeness, 
appropriate interaction stiles, negotiation, etc., 
“Fingerspitzengefiihl” (Forss and others, 2002, 
35), tact (van Manen, 1991) or adequate rela¬ 
tionship (Abma, 2006) in social interaction, 
etc. Concepts like the deliberative democratic 
approach promoted by House (1980; House & 
Howe, 2000; etc.), the fourth generation evalu¬ 
ation (Guba & Lincoln, 1989) or empower¬ 
ment evaluation (Fetterman and others, 1996) 
focus particularly on this issue. 

The values addressed in this domain are 
the topic of social psychology and focus on the 
question of optimizing social interaction, the 
overarching value being that the dignity, the 
personality, the needs and the autonomy of the 
partners are respected while the necessary in¬ 
formation is acquired. 
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4. Teisines vertybes apima visas jstatymo 
ir asmenij, turinciij priversti jj vykdyti, nu- 
matytas vertybes (pvz., administravimq). Si 
sritis, ko gero, maziausiai aptariama atliekant 
vertinimo tyrimus. Visq pirma ji nagrineja- 
ma kilus tam tikroms problemoms, pavyz- 
dziui, vyriausybes atstovams pareikalavus, 
kad biiti} finansuojami tik atsitiktinai atrinkti 
kontroliuojami eksperimentai (House, 2003; 
Phillips, 2006). 

Ekspertai turi buti susipazin^ su pagrin- 
dinemis jvertinimo teisinemis normomis ir 
zinoti, kokias pasekmes jq sprendimas gali 
tureti institucijai. |vertinimo komitetas gana 
daznai skirtingai interpretuoja to paties jsta¬ 
tymo teksUj, o tinkamq interpretacijq reikia 
paaiskinti. Be to, privalu atskirti teisinius nu- 
rodymus ir politinius poreikius - pirmieji yra 
virsesni. Galiausiai butina nepriklausomybe 
nuo jgaliojancios jstaigos, kad butq isvengta 
jtakos sprendimui. 

5. Ekonomines vertybes - tai vertybes, 
susijusios su jvertinimo finansiniu ir dar- 
buotojij aspektu arba laiko istekliais, jskai- 
tant sqnaudas, skirtingq socialiniq dali- 
ninku skirtas pastangas, isorinius isteklius, 
medziagas, technin^ ir programing jrangq 
ir pan. Paprastai publikacijose apie jvertini- 
mq neminimas biudzetas, nebent aptariant 
bendrus suvarzymus (House, 2003). Cia ne- 
priklauso institucijos islaidrj ir gautos nau- 
dos bei ekonomiskumo jvertinimas, nes jie 
susijf su tiriamos institucijos finansinemis 
s^lygomis, o ne su vertinimo procesu. Atsa- 
kingos salys turi apskaiciuoti ir susitarti del 
islaidp pries pradedamos konkretp jvertini¬ 
mo process 

Svarbus klausimas - ekspertp honoraras. 
Kartais jj moka jgaliojanti jstaiga, kartais pa- 
dengiamos tik keliones ir apgyvendinimo is- 
laidos. Nuo to priklauso, kiek laiko ekspertas 
ketina skirti apsilankymui vietoje ir ataskaitai 
parasyti. Kalbant apie islaidas, reikia jvertin- 
ti ir darbo kruvj, ypac rengiant jsivertinimo 
ataskait^. 


As all social interactions, appropriate be¬ 
havior is situation specific (see Patry, 2011), 
which means that the evaluators must take into 
account the contextual conditions. In particu¬ 
lar they must be aware that they have a differ¬ 
ent role in the evaluation process than in usual 
interaction; this is the more important if the 
peers meet a person from the institution to be 
evaluated at other occasions, which happens 
frequently in small countries like Austria, Swit¬ 
zerland and Lithuania. 

4. Legal values include all values provided 
by the law and by those who are supposed to 
enforce it (e.g., administration). This domain 
is probably least discussed in evaluation re¬ 
search, and it is addressed primarily when 
particular problems arise such as the require¬ 
ments by governmental representatives that 
only randomized controlled experiments will 
be funded (e.g., House, 2003; Phillips 2006). 

The peers must be aware of the legal rules 
underlying the evaluation and of the conse¬ 
quences their judgment might have for the in¬ 
stitution. Different interpretations of the same 
legislative text within the evaluation committee 
are quite frequent, and the appropriate inter¬ 
pretation needs to be clarified. It is also neces¬ 
sary to distinguish between legal prescriptions 
and political needs, the former having priority 
over the latter. And finally, the independence 
from the mandatory agency to avoid it influ¬ 
encing the judgment is indispensable. 

5. Economic values concern all the values 
connected with the financial and personnel 
background of the evaluation or the time re¬ 
sources, including the costs, the efforts to be 
spent by the different stakeholders, external 
resources, material, hard- and software, etc. 
Budgetary issues are usually not addressed in 
publications on evaluation, unless general re¬ 
strictions are discussed (e.g. again in House, 
2003). It must be reminded here that evalua¬ 
tions of the institutions cost-benefits and cost- 
effectiveness are not among these issues since 
they address the financial background of the 
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6. Asmenines vertybes apima visus socia- 
liniams dalininkams, vertintojams ir kitiems 
asmeniskai svarbius dalykus. Aprepiama ir 
rizika, kuri^ socialinis dalininkas gali susieti 
su jvertinimu, t. y. vertybes, kurioms jver- 
tinimas gali buti zalingas (pvz., neigiamas 
jvertinimas gali sukelti pavojij institucijos 
vadovams). Tai apima ir konkrecias vertin- 
toji} vertybes, nepatenkancias j kitq. sritj, pa¬ 
vyzdziui, asmenines konkreciij socialinii} da- 
lininkij savybes ir isplaukiancias antinomijas 
nagrineja E. Chelimsky (2006) ir H. Simons 
( 2000 ). 

Kai kuriais atvejais ekspertai suintere- 
suoti sumenkinti vert in amt), institucij;}, kad 
geriau atrodyti} jij institucija. Be to, vertin- 
tojai turi atsizvelgti j tai, kad vertinamieji 
gali tureti ypatingij interest; pavyzdziui, 
vertinamasis del darbo kreipesi j institucija, 
is kurios buvo kviestasis ekspertas. Bet ko- 
kiu atveju, vertinamieji nori atrodyti kuo ge¬ 
riau - bendraudami visi taip elgiames (Bau- 
meister, 1982). 

7. Visuomenineje ir politineje vertybiit 
srityje kalbama apie visas temas, kurias ver- 
tinimo metu paliecia politika. Sios vertybes 
virsija socialinii} dalininki} grupii} ribas ir 
taikomos visuomenei arba visai saliai, arba 
specialioms grupems, pavyzdziui, mokesciij 
moketojams, mazumoms, spaudimo gru¬ 
pems ir pan. (pvz., vyriausybiniu program^ 
vertinimas visada turi visuomeninj ir politinj 
poveikj, nes kalbama apie zmonii} gerov^ bei 
mokesciij moketojij pinigus, kartais ir poli- 
tikiy perrinkim;}). Tai ypac gerai rodo nese- 
niai atlikti PISA ir TIMS tyrimai; jie nelabai 
informatyvus (Loveless, 2007), taciau daug 
politikij naudojasi pasirinktais duomenimis 
savo megstamiems projektams paremti (Pat- 
ry, 2008). 

Pagal jvertinimo samp rati} susidaro 
skirtingos pozicijos: vieni ekspertai pritaria 
islaisvinanciam ar mokejimi} suteikianciam 
vertinimui, kiti rupinasi konkreciomis gru- 
pemis (pvz., daznai gincijamasi del lygii} 


institution under investigation and not of the 
evaluation process. In an evaluation the costs 
must be estimated and agreed upon between 
the responsible parties before the concrete 
evaluation process is started. 

One important issue is the honorary of the 
peers. In some cases an honorary is paid by the 
mandating agency, in others only the travelling 
and accommodation costs are paid. Whether 
an honorary is paid or not might have an im¬ 
pact on the time a peer is willing to invest in 
preparing the site visit and in writing the re¬ 
port. Among the costs one must also consider 
the workload of the evaluees, particularly for 
the self-evaluation report. 

6. Personal values deal with everything that 
is personally important to the stakeholders, the 
evaluators, etc. It includes the risk a stakeholder 
might link with the evaluation, i.e., his or her 
values that might be challenged or jeopardized 
by the evaluation (e.g., a negative outcome of 
the evaluation might be threatening for those in 
a leading position in the institution). It includes 
also the specific personal values of the evaluators 
which do not belong to another area. The per¬ 
sonal values of particular stakeholders and the 
ensuing antinomies are addressed, for instance, 
by Chelimsky (2006) and Simons (2000). 

In some cases peers had the interest to put 
down the institution they are evaluating so that 
their own institution looks better. The evalua¬ 
tors must also take into account that the eval¬ 
uees might have special interests; for instance 
in one case an evaluee had applied for a job 
in the institution the peer came from. And in 
any case the evaluees are likely to be interested 
in looking as good as possible, as we all do in 
social interactions (Baumeister, 1982). 

7. Societal and political area values deal 
with all topics touched by politics in evaluation- 
processes. These values refer to groups beyond 
the particular stakeholders, such as the society 
or a country as a whole, or specific groups like 
taxpayers, minorities, pressure groups, etc. 
Evaluations of governmental programs, for 
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lyciij galimyhiij) ir pan. Daznai sie priori- 
tetai buna numanomi, bet gali buti ir labai 
galingi. 

Sriciq sqrasas gana sutartinis, gana su- 
tampantis. Pavyzdziui, 1 (etines vertybes) 
ir 3 (socialines ir interaktyvios vertybes) 
sritys turi daug bendrumij. Kai kurie au- 
toriai, pavyzdziui, E. Guba ir Y. S. Lincoln 
(2000), gali gincytis, kad jq metateorines 
vertybes (t. y. naturalus pavyzdys), cia lai- 
komos metodologinemis, buvo pasirinktos 
del etiniq priezasciq, todel irgi gali buti 
priskirtos pirmajai sriciai - etinems ver- 
tybems. Siame darbe siekiama ne klasifi- 
kuoti nedviprasmiskai, o sukurti strukturq, 
galinciq padeti nustatyti galimas vertybes; 
tiksliau, stengiamasi parodyti vis/}, galimij 
vertybiij, svarbiij antagonizmams, spektrq. 
Sritys perzengia etikos ribas, kurias apta- 
ria M. Morris (2005, 2007), M. Morris ir 
R. Cohn (1993), D. Turner (2003) ir kiti. 
Pavyzdziui, M. Morris (2005) daugiausiai 
demesio skiria informacijai tvarkyti (meto- 
dine sritis) ir poziuriui, kaip elgtis su so- 
cialiniais dalininkais (asmeniniq vertybiq 
sritis), bei sip dviejq sriciq sandurai, pavyz¬ 
dziui, „Socialinis dalininkas daro spaudimq 
jvertintojui pakeisti rezultatq pateikimq" 
(p. 132). 

Antinomijos gali egzistuoti ir vienoje 
srityje; pavyzdziui, Herrmann (1976) me- 
todq problem^ pavadino diapazono apim- 
ties tikslumo dilema (mes jq vadiname an¬ 
tinomy a): kadangi vertinimo instrumentas 
negali virsyti tam tikro ilgio, vertintojas 
turi nustatyti pusiausvynj tarp (1) kuo dau- 
giau charakteristikp jvertinimo (diapazono 
apimtis) ir (2) kuo patikimesnio jp jverti¬ 
nimo (tikslumas). Kuo placiau vertinamos 
specifines charakteristikos, tuo patikimes- 
ni buna jvertinimai, taciau taip vertinamp 
charakteristikp buna maziau. Vykstant eks- 
pertiniam vertinimui, ekspertai turi ribottj 
laik^, todel atsiranda antinomy a, ar issa- 
miau vertinti vien§ tem;j ir gauti daugiau 


instance, have always a societal and political 
impact since human welfare as well as taxpay¬ 
ers’ money (and sometimes also the politicians’ 
re-elections) are at stake. This is particularly 
visible in the recent PISA and TIMS studies, 
from which little can be deduced (Loveless, 
2007) yet many politicians use the data of their 
choice from these studies to support their pet 
project in a simplistic way against their oppo¬ 
nents’ concepts (Patry, 2008). 

Depending on the concept of evaluation, 
different stances can be taken. Some peers 
might adhere to an emancipatory or empow¬ 
erment evaluation, others might care particu¬ 
larly for particular groups (for instance equal 
opportunity with respect of sex is often an is¬ 
sue), etc.; often these priorities are implicit and 
not expressed within the evaluation commit¬ 
tee but they can be quite powerful. 

This list of domains is somehow arbitrary, 
and there is considerable overlap between the 
domains. For instance, the domains 1 (ethical 
values) and 3 (social and interactional values) 
have many communalities. Some authors like 
Guba and Lincoln (2000) might also argue 
that their meta-theoretical values (i.e., the nat¬ 
uralistic paradigm) which have been classified 
here as methodological here has been chosen 
on ethical reasons and therefore should belong 
to the first domain, ethical values. The idea 
here is not to give an unambiguous classifica¬ 
tion, but to provide a framework that might be 
helpful to identify possible values; in particular, 
it is presented here as an attempt to show the 
full scope of possible values which might play 
important roles in antagonisms. The domains 
go beyond the ethical challenges discussed by 
Morris (2005,2007), Morris and Cohn (1993), 
Turner (2003) and others. Morris (2005), for 
instance, focuses mainly on how to deal with 
information (methodical domain) and with 
stakeholders (domain of personal values) and 
between these two domains, such as “The 
evaluator is pressured by a stakeholder to alter 
presentation of the findings” (p. 132). 
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patikimos informacijos, ar nagrineti kelias 
temas ir gauti ne tokiq patikimq informaci- 
j;j. Kita vertus, antinomijos gali egzistuoti 
ir tarp sriciij. Pavyzdys gali buti S. Dela- 
montAS (2005) - jis isskyre praktin^ ver- 
tyb^ (klasifikuotinq kaip priklausanciq 7 - 
visuomeniniq ir politiniq vertybip - sriciai, 
nes politikai ir visuomene domisi prakti- 
nemis jvertinimo pastangp pasekmemis), 
mokslinj reiksmingump (vertintojo asme- 
ninis interesas, 6 sritis, formuluojama kaip 
vertintojo interesas vel buti paprasytam 
atlikti i vert in imp), pasmerktp zlugti pro- 
jektp (del metodologinip trukump, 2 sritis) 
ir „nezinomp pasaulip stebuklp studijavi- 
mp“ (vel asmeninis interesas: noras pazinti 
institucijp). 


3 

ALTERNATYVIOS IR PALAIPSNINES 
ANTINOMIJOS 

Galima isskirti du antinomijp tipus, kuriuos 
pavadinsime alternatyviomis ir palaipsnine- 
mis antinomijomis (zr. 1 lentel^). Terminas 
dilema paprastai vartojamas alternatyvipja 
prasme, t. y. protagonistas gali atlikti P arba 
P 2 , bet ne jp kombinacijp. Tipiskas pavyzdys - 
M. Scriven (1974) siulomas skirtumas tarp 
tikslp turincio ir jo neturincio jvertinimo: 


There can be antinomies within one domain; 
within the methods, for instance, there is the 
problem that Herrmann (1976) has called the 
bandwidth-hdelity dilemma (we call it an antin¬ 
omy): given that an assessment instrument must 
not exceed a certain length, an evaluator must 
find the right balance between (1) assessing as 
many characteristics as possible (bandwidth) and 
(2) assessing them as reliably as possible (fidel¬ 
ity); the longer the assessment for a specific char¬ 
acteristic, the more reliably one can assess it but 
the less such characteristics can be assessed. This 
can be transferred to peer evaluation: the peers 
have only a limited time available during their 
site visit, and the antinomy is whether to deepen 
the assessment on one topic, getting more reli¬ 
able information, or to address several topics but 
getting information that is less trustworthy. On 
the other hand, there can be antinomies between 
domains. An example is Delamont (2005) who 
distinguished practical value (which we would 
classify as belonging to domain 7: societal and 
political domain because politicians and society 
are interested in practical consequences of the 
evaluation endeavor), scholarly immorality (the 
evaluators personal interest, domain 6; we would 
reframe this as the evaluator s interest to be asked 
again to do an evaluation), project doomed to fail 
(due to methodological flaws, domain 2), and ex¬ 
plore wonders of unknown worlds (again, per¬ 
sonal interest: curiosity about the institution). 

3 

ALTERNATIVE AND GRADUAL 
ANTINOMIES 

One can distinguish two types of antinomies 
which we call alternative and gradual antino¬ 
mies (cf. the columns in table 1). The term 
“dilemma” is mostly used in the alternative 
sense, which means that the protagonist can 
do either P or P , but no combination of both. 
A typical example is Scriven’s (1974) distinc¬ 
tion of goal-based versus goal-free evaluation: 
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galima atsizvelgti arba neatsizvelgti j jvertini- 
mo tikslus, nes „tikslij aptarimas ir jvertini- 
mas (gali buti) nereikalingas, gal net klaidi- 
nantis zingsnis" (p. 35). 

Kitp antinomijij nesuderinamumas 
nera grieztas. Taciau dvi vertybes buna de- 
rinamos: santykiai tarp P ir P 2 priklauso 
tipui daugiau P, maziau P,; mes tai vadi- 
name palaipsninemis antinomijomis, nes 
yra svarbu jvykdymo kiekybe (pvz., pagal 
vertybes realizavimo procentq) - galimas 
kompromisas. Kaip pavyzdj panaudosime 
diapazono apimties tikslumo antinomijij 
(Cronbach, 1990, p. 208): musq darbe si 
antinomija reiskia, kad per tam tikrij ribo- 
tq laikq galima suderinti nagrinetinq temp 
skaiciij (diapazono apimtis) ir gautos infor- 
maeijos tikslumij bei issamumij (tikslumas). 
Esme - ne pasirinkimas tarp „diapazono 
apimties ir tikslumo", veikiau - „kuo dides- 
ne diapazono apimtis, tuo zemesnis tikslu¬ 
mas", todel vertintojas turi nusprqsti, kiek 
reikia sumazinti tikslumq, kad padidetq 
diapazono apimtis. 

Atpazintq problem^ L. J. CronbachAS 
(ten pat) pavadino diapazono apimties ir 
tikslumo dilemomis. Pavyzdys rodo, kad di- 
lema kartais naudojama palaipsninems anti¬ 
nomy oms; siuo aspektu terminas vartojamas 
ir sio darbo pavadinime. Alternatyviij ir pa- 
laipsninip antinomijij isskyrimas turi didel^ 
praktin^ reiksm^, nes antinomijos sprendi- 
mas priklauso nuo to, ar ji palaipsnine (tada 
jmanomas kompromisas), ar alternatyvi (rei¬ 
kia pasirinkti tarp nesuderinamij vertybiij). 

4 

ANTINOMIJOSTIKSLO IR PRIEMONiy 
LYGMENIMIS 

Dviejij vertybiij antinomija gali buti tikslo 
arba priemoniplygmenimis (1 lentele). Tikslo 
lygmeniu antinomija tarp tikslp ir G, gali¬ 
ma suformuluoti taip: 


One can either consider or not consider goals 
in evaluations since “consideration and evalua¬ 
tion of goals (can be) anD unnecessary but also 
a possibly contaminating step” (p. 35). 

In other antinomies the mutual exclusion 
is not strict, instead there is a trade-off between 
the two values: the relationship between P and 
P 2 is of the type the more P, the less P,; we call it 
gradual antinomies because the quantity of ac¬ 
complishment (for example, in percent of the 
full realization of the value) is at stake: a compro¬ 
mise is possible. An example is the bandwidth- 
fidelity antinomy (Cronbach, 1990, p. 208): in 
our adaptation this antinomy means that within 
a given restriction in time, there is a trade-off 
between the number of topics that can be ad¬ 
dressed (bandwidth) and precision and depth 
of the information that has been gained (fidel¬ 
ity). The issue is not “either bandwidth or fidel¬ 
ity” but rather “the broader the bandwidth, the 
lower the fidelity”, and the evaluator must decide 
how much to take into account the reduction of 
fidelity in order to enhance bandwidth. 

As the example of the bandwidth-fidelity 
dilemma , as Cronbach (loc. cit.) called the 
problem he had recognized, shows, the term 
dilemma is sometimes used for gradual antin¬ 
omies; it is in this sense that the term is used 
in the title of the present contribution. The 
distinction between alternative and gradual 
antinomies is of great practical importance 
since the solution of an antinomy will depend 
on whether it is gradual (then a compromise is 
possible) or alternative (a choice between the 
mutually exclusive values must be made). 

4 

ANTINOMIES ON THE GOAL AND 
ONTHE MEANS LEVEL 

The antinomy of two values can be either on the 
goal level or on the means level (rows in table 1). 
On the goal level, the antinomy between the 
goals G and G can be formulated as follows: 
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1 lentele. Dvi antinomijq taksonomijos dimensijos 
Table 1 . Two dimensions of the taxonomy of antinomies 



ANTINOMIJOS TIPAS / TYPE OF ANTINOMY 

ALTERNATYVI /ALTERNATIVE 

PALAIPSNINE / GRADUAL 

Antinomijos 

vieta 

Location 
of the 
antinomy 

Tikslai 

Goals 

Tikslai G ir G 2 vienas kitp paneigia. 

Goals Gj and G, are mutually 
exclusive. 

Kuo labiau realizuojamas G , tuo maziau 
realizuojamas G 2 . 

The more G is realized, the less can G 2 be 
realized. 

Priemones 

Means 

Tikslai suderinami; priemones M 
ir M 2 viena kitp paneigia. 

Goals are compatible; Means M 
and M, are mutually exclusive. 

Tikslai suderinami; kuo labiau realizuojama 
Mj, tuo maziau gali buti realizuota M . 

Goals are compatible; the more M is 
realized, the less M, can be realized. 


1. vertintojas taiko j tikslus Gj ir G 2 ; 

2. Gj aprasyta situacija butp susidary- 
sianti, jeigu buvo pasiektas tikslas G ( - negali 
jvykti vienu metu su G, aprasyta situacija. 

Taigi vertintojas gali kaip vertingus ginti 
abu tikslus - Gj ir G 2 , bet nejmanoma abiejp 
pasiekti vienu metu. Pavyzdys is vertinimo 
atvejo: vertintojas uztikrino vertinamiesiems 
konfidencialump, bet administracija grasina 
sankcijomis, jeigu sis neatskleis pareiskimp 
autorip tapatybes. vertintojas noretp paten- 
kinti abi puses, taciau tai nejmanoma - situ¬ 
acija, kuricj apraso tikslas respondentai neat- 
skleidziami (Gj), nesuderinama su situacija, 
kuriq apraso tikslas respondentia tapatybe yra 
zinoma administracijai (G 2 ), nes pirmasis lo- 
giskai paneigia antrqjj. 

Antinomijos priemonii} lygmeniu atveju 
vertintojas tam tikroje situacijoje vel susidu- 
ria su dviem tikslais Cy ir G 2 , taciau ir G, 
aprasytos situacijos viena kitos nepaneigia. 
Vis delto, noredamas pasiekti G , protagonis- 
tas mano, kad butp tinkama imtis veiksmij 
(priemonip) M , o G 2 pasiekti laiko reika- 
linga M , bet M ir M 2 negalima atlikti vienu 
metu: nera veiksmo (ar veiksmp rinkinio), 
leisiancio pasiekti ir G , ir G,. Pavyzdziui, 
tureti galiojancius duomenis (G ) ir mazq 
biudzetp (G ) logiskai nepriestarauja vienas 
kitam. Taciau daugeliu atveju, norint gauti 
aukstos kokybes duomenis, reikia daug dar- 
bo, uz kurj turi buti sumoketa (M ), o norint 


1. The evaluator aims at the goals G and G v 

2. The situation described by G - that is, 
the situation that would result if the goal G 
was achieved - cannot occur simultaneously 
with the situation described by G 2 . 

Hence the evaluator might defend both 
goals, Gj and G 2 , as valuable, but the achieve¬ 
ment of both of them simultaneously is impos¬ 
sible. As an example from evaluation, take an 
evaluator who has warranted confidentiality to 
the evaluees but is requested - and threatened 
with sanctions by the administration to reveal 
the identity of the authors of these statements. 
The evaluator would like to satisfy both, but 
this is impossible because the situation de¬ 
scribed by the goal the respondents remain un¬ 
disclosed (Gj) is incompatible with the situation 
described by the goal the respondents’ identity 
is known to the administration (G ), the former 
being the logical negation of the latter. 

In an antinomy on the means level, the 
evaluator has, again, the two goals Gj and G 2 in a 
particular situation, but the situations described 
by Gj and G, are not mutually exclusive. Howev¬ 
er, to achieve Gl, the protagonist thinks that the 
action (mean) would be appropriate, while to 
achieve G 2 he or she believes M 2 would be neces¬ 
sary, and Mj and M 2 cannot be performed simul¬ 
taneously: There is no action (or set of actions) 
that would permit to achieve both Gj and G 2 . 
For instance, having valid data (G ) and having 
a low budget (G 2 ) is not logically contradictory. 
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islaikyti mazcj biudzetq, reikia kuo maziau 
isleisti (M 2 ). 

Priimant sprendimq, ar egzistuoja anti- 
nomija tikslq arba priemoniij lygmenimis, 
reikia jvertinti, ar situacijos, kurias apraso 
tikslai, yra suderinamos. Tikslij antinomijoje 
jos nesuderinamos; priemoniij antinomijoje 
galima (bent is principo) jsivaizduoti prie- 
mones M , patenkinancias ir G , ir G 2 . Mine- 
tame pavyzdyje galima atlyginti ne pinigais - 
pavyzdziui, dalj darbo atlieka apmokyti 
bakalauro studijij studentai ir magistrantai 
(Morrow ir kt., 2006) arba savanoriai (Cam- 
berg ir kt., 2006). 

Abi antinomijos priemoniij ir tikslij lyg- 
menyse gali buti alternatyvios arba palaipsni- 
nes (1 lentele). 


5 

PRINCIPg HIERARCHIJA 

Etiniu poziuriu, vertintojij nagrinejamos 
vertybes nera ekvivalentiskos. Galima pri- 
taikyti keletij principij, kuriais nustatomas 
vertybiij prioritetas. Mes siulome devynis. 
Jie grindziami etikos teorija, kuri isplaukia 
is J. A. Rawl (1971) Teisingumo teorijos, ta- 
ciau cia nejmanoma pateikti sijsajij tarp sios 
teorijos ir miisij principij (pvz., kai kuriuos 
argumentus galima rasti House, 1976 ir 
Morris, 2003). 

Svarbu aiskiai atskirti vertybes ir prin- 
cipus. Musi} aptartos vertybes - tai dalykai, 
kuriuos zmones laiko svarbiais. Jos gali buti 
ir nebuti pagrjstos; pavyzdziui, asmenines 
vertybes (pirmosios taksonomijos dimensi- 
jos 6 sritis) retai kada pagrjstos, o etines ver¬ 
tybes (1 sritis) daznai yra aiskiai motyvuo- 
tos. Cia aptariami principal - normatyviniai 
siulymai vertintojams, kaip priimti sprendi- 
m;j vertybiij antinomijij atveju. Ir patys prin- 
cipai gali buti antinominiai, tad svarbu, kad 


In most cases, however, providing high quality 
data requires much work which must be paid for 
(M ) but keeping up with the low budget requires 
spending as little as possible (M ). 

For the decision whether there is an an¬ 
tinomy on the goals or on the means level one 
must look at the compatibility of the situations 
described by the goals. In the goals antinomy, 
these situations are incompatible; in a means 
antinomy, one can, at least in principle, imag¬ 
ine a means M 3 that satisfies both G and G . In 
the above example, this could be another type 
of reward than money - for example, some of 
the work is done by trained undergraduates or 
graduate students (Morrow and others, 2006) 
or by volunteers (Camberg and others, 2006). 

Both antinomies, on the goals and on the 
means level, can be either alternative or gradu¬ 
al, as shown in table 1. 

5 

HIERARCHY OF PRINCIPLES 

From an ethical point of view, the values eval¬ 
uators have to deal which are not equivalent. 
Several principles might be applied to establish 
a priority of values. We propose nine such prin¬ 
ciples. These principles are based on an ethical 
theory that follows Rawls’ (1971) Theory of 
Justice, but it is not possible here to give an ac¬ 
count of the links between this theory and our 
principles (some arguments can be found, for 
instance, in House, 1976, and Morris, 2003). 

It is important to make a clear distinction 
between the values and the principles. The val¬ 
ues discussed above are issues which people 
think are important. They may or may not be 
justified; personal values (domain 6 in the first 
dimension of the taxonomy, see above) for 
instance are rarely justified, whereas ethical 
values (domain 1) usually are explicitly argued 
for. The principles discussed here are norma¬ 
tive suggestions to evaluators how to decide 
in the case of antinomies of values. Since the 
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jie butq hierarchiniai: jeigu du arba daugiau 
principle yra antinominiai (t. y. pagal x prin- 
cipq, a vertybe turi prioritetq, o pagal y prin- 
cipq, b vertybe turi prioritetq), auksciausio 
laipsnio principas turi prioritetrj. (1 laipsnis 
yra auksciausias), t. y. 1 laipsnio principas 
virsesnis uz 2 laipsnio principq ir pan. Kitaip 
sakant, kuo aukstesnis laipsnis, tuo stipres- 
nis imperatyvas vadovautis principu. Pasak 
N. Gross ir kolegq (1958), vaidmenq teorijo- 
je galima atskirti privalomus, preferencinius 
ir leistinus lukescius: kuo aukstesnis laips¬ 
nis, tuo labiau privalomas reikalavimas, siuo 
atveju - principas. 

Galima pamineti pavyzdj: etiniq dalykij 
(2 principas) negalima paaukoti del meto- 
diniq priezasciq (8 principas); tai reiskia, 
kad, pavyzdziui, draudziama pakenkti sub- 
jektams, - nors tai laikoma neetiska, - net 
jei tuo butq galima sustiprinti jvertinimo 
rezultatij pagrjstumq. Hierarchija pateikia- 
ma kaip pasiulymas, tai - tolesniij diskusijq 
dalykas. Tarkime, kad 1-3 laipsnio prin¬ 
cipal yra gana tvirti. Kalbant apie likusius 
principus jmanoma, jog skirtingij jvertini¬ 
mo pavyzdziq vertintojai (pvz., skirtingos 
jvertinimo medzio sakos, Alkin, Christie, 
2004) pripazjsta skirtingus prioritetus; sq- 
rasq galima naudoti kaip pagrindq toliau 
detalizuojant prioritetus (principal apiben- 
drinti 2 lenteleje). 

1. Svarbiausias principas - bet kokiq ver- 
tybiij prioritetas turi buti jrodomas, argumen- 
tuojamas ir aiskiai pagrindziamas, kad butq 
suprantamas skirtai auditorijai. Mes tvirtina- 
me, kad vertinimo isvada yra priimtina, kai 
teiginiai pagrindziami (Dewey, cituojama is 
Phillips, Burbules, 2000, p. 31; kalbama apie 
pagrjstus teiginius). Vertintojcj nuomone del 
teiginiij pagrjstumo gali skirtis, bet neginci- 
jama, kad to reikia siekti. Vis delto bet koks 
argumentavimas yra ribotas ir tobulij teigi- 
nip nebuna. Vertintojai diskutuoja pagal savo 
galimybes, atsizvelgdami j visas problemas. 
Be to, jvertinimo ataskaitos erdve taip pat 


principles themselves may be in antinomy, it is 
important that they are hierarchical: If two or 
more principles are in antinomy (i.e. accord¬ 
ing to principle x, value a has priority, whereas 
according to principle y, value b has priority), 
the one with the higher rank (with rank 1 as 
highest rank) has priority, i.e. the principle 
with rank 1 prevails over the one with rank 2, 
etc. In other words: the higher the rank, the 
stronger is the imperative to follow the princi¬ 
ple. With Gross et al. (1958) one can apply the 
distinction between mandatory, preferential, 
and permissive expectations in role theory: the 
higher the rank, the more mandatory is the re¬ 
quirement, in this case the principle. 

To mention just one example: ethical issues 
(principle 2) must not be sacrificed for me¬ 
thodical reasons (principle 8); this means, for 
instance, that harming the subjects - which is 
considered as unethical - is prohibited even if 
the validity of the evaluation findings could be 
enhanced with such procedures. The hierarchy 
is presented as a proposition that is subject to 
further discussion. We assume that the prin¬ 
ciples from rank 1 through rank 3 are quite well 
established. For the remaining principles, it is 
possible that evaluators from different para¬ 
digms in evaluation (e.g., the different branches 
in the evaluation tree, Alkin & Christie, 2004) 
have different priorities; the list then can be 
used as base for making the priorities explicit. 

The principles are summarized in table 2 
and will be explained in detail below. 

1. The most important principle is that 
any values priority must be arguable, must be 
argued for and must be justified explicitly in 
such a way that the intended audience can un¬ 
derstand it. We claim that a conclusion from an 
evaluation is only acceptable if the assertions 
are warranted (Dewey, quoted from Phillips & 
Burbules, 2000, 31, speaks of warranted assert- 
ibility). Evaluators might disagree about how 
warranted assertibility is achieved, but that it 
must be aimed at is uncontested. However, any 
argumentation has its limits, and it is impossible 
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2 lentele. Principq hierarchija 
Table 2. Hierarchy of principles 


LAIPSNIS 

RANK 

PRINCIPAS 

PRINCIPLE 

PAAISKINIMAS 

EXPLANATION 

1 

Garantuotas 

tvirtinimas 

Warranted 

assertibility 

Teiginiai turi buti argumentuoti ir suprantami visiems socialiniams dalininkams. 
Statements must be argued for and made transparent to all stakeholders. 

2 

Etines vertybes 

Ethical values 

Reikia gerbti etines normas, socialines vertybes ir asmeninius vertintojq 
jsitikinimus. 

Ethical standards, social values and the evaluators’ personal conviction must be 
respected. 

3 

Tikslai auksciau 
uz priemones 

Goals over 

means 

Priemones reikia pasirinkti pagal tikslus (jskaitant neigiamo salutinio poveikio 
vengimq), kuriuos butina pasiekti ir kurie patys nera vertingi. 

Means must be chosen in function of the goals (including avoiding negative 
side-effects) that should be achieved and have no value by themselves. 

4 

Gerove 

Welfare 

Reikia atsizvelgti j visq subjektq (ne tik zmoniq), kuriuos veikia arba gali paveik- 
ti jvertinimas, gerov§. 

Welfare of all organisms that are or may be affected by the evaluation (not only 
humans) must be taken into consideration. 

5 

Atsizvelgiama j 
visas susijusias 

Consider all 
relevant values 
vertybes 

Vertintojas turi aktyviai identifikuoti susijusias vertybes (pvz., jskaitant sociali- 
niq dalininkq, kurie neatskleidzia savo interest}, vertybes) ir j jas atsizvelgti. 

The evaluator must actively identify relevant values (including, for instance, 
values of stakeholders who do not make explicit their interests) and take them 
into account. 

6 

Teisine sritis 

Legal domain 

Turi buti vertinama pagal jstatymq, taikomq atitinkamam atvejui 

Evaluations must be performed within the limits of the law that apply to the 
particular case. 

7 

Biudzetas 

Budget 

Reikia laikytis sutarto biudzeto; pirmiausia apskaiciuojamos vertinimo islaidos, 
kurias galima pakeisti tik susitarus. 

The budget that was agreed upon at the beginning of the evaluation process 
must be kept; the costs of the evaluation must be estimated at the beginning, 
and only when an agreement has been reached can it be changed. 

8 

Metodai 

Methods 

Reikia naudoti tik labiausiai tinkamus metodus: pirma, planavimas ir vertini- 
mas, antra - statistika. 

Within the possibilities, the best suited methods must be used. This is, first, an 
issue of design and assessment, and statistics come only second. 

9 

Asmenines so- 
cialiniq dalinin- 
kt} vertybes 

Personal 
values of the 
stakeholders 

Reikia tinkamai apsvarstyti socialiniq dalininkq vertybes. 

The stakeholder’ values must be considered appropriately. 


ribota, kaip ir auditorijos, kuriai skiriami ar- 
gumentai, kompetentingumas (pvz., detalus 
metodiniai argumentai turi prasm^ tik tada, 
jeigu adresatai yra jgij§ reikalingij metodiniij 
baziniij ziniq). Taigi vertintojas tures daryti 


to achieve any form of perfect assertibility. Eval¬ 
uators are constrained to argue within their 
possibilities, being aware of all the problems 
which are related with this. Further, there are 
space restrictions in the evaluation report, and 
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kompromisq tarp nuodugnaus argumenta- 
vimo ir atitinkamai auditorijai pateikiamos 
informacijos kiekio. Taciau net jeigu teiginys 
nejtrauktas j ataskaitq, vertintojas paprasytas 
turetq pateikti pagrjstus argumentus. 

Sio principo pateisinimas - tai prielaida, 
kad racionalus argumentavimas, aprepiant 
visus prieinamus duomenis (jskaitant galio- 
jancius empirinius duomenis, bet neapsiribo- 
jant jais), yra geriausias tinkami} sprendiimj 
pagrindas, nors ir negali pakeisti sprendiimj 
priemimo. Zvelgiant jvertinimo teorijos as- 
pektu, tai ir yra priezastis, kodel jvertinimas 
apskritai atliekamas. 

Sis principas visiskai suderinamas su de- 
mokratinio jvertinimo reikalavimu, pateiktu 
E. R. House ir K. Howe (2000), ir susij^s su 
trimis reikalavimais: jtraukimo (atsizvelgti j 
visus susijusius interesus; op. cit„ 5f.), dialogo 
(atrasti realius interesus per dialogq; op. cit., 
6£), ir apsvarstymo (vertybes neturi buti pri- 
imamos kaip negincytinos; jas reikia isnagri- 
neti per racionalij process op. cit., p. 8f.). Mi- 
netq autorip pateiktas pagrindimas gali buti 
pritaikytas cia. 

2. Antras principas reikalauja gerbti 
etines vertybes (pirmos taksonomijos di- 
mensijos 1 sritis); joms galima priskirti kai 
kurias vertybes is socialines ir interakty- 
viosios sriciq (3 sritis). Esant antinomijai 
etines srities viduje, reikia surasti ir jro- 
dyti prioritetq pagal 1 principq. Jeigu etine 
vertybe priestarauja kitos srities vertybei, 
vertintojas tik del rimtq priezasciq gali pir- 
menybes (palaipsnines antinomijos atve- 
ju - labai svarbu) neteikti etinems. Arba 
svarstyti galimyb^ visai nevertinti (zr. Che- 
limsky, 2006). Vadinasi, vertintojai nedaro 
nieko, kas priestarauja jq jsitikinimams, ir 
jp negalima versti. 

Vertybes, kurias iskelia socialiniai da- 
lininkai (arba kurias galetp iskelti; zr. 5 ir 9 
principus), nelaikomos imperatyviomis; jos 
veikiau vertinamos kaip svarstytinos ver¬ 
tybes; skirtingoms vertybems priskiriamas 


the competence of the audience is limited (for 
instance, detailed methodical arguments make 
sense only if the addressees have the necessary 
methodical background knowledge). Hence 
the evaluator will have to make a compromise 
between in-depth argumentation and amount 
of information provided to the respective au¬ 
diences. But even if the argument does not ap¬ 
pear in the report the evaluator should be will¬ 
ing and able to present appropriate arguments 
upon request as far as possible. 

The justification for this principle is the as¬ 
sumption that rational argumentation capital¬ 
izing on all information available (including, 
but not restricted to, valid empirical data) is the 
best base for good decisions, although it can¬ 
not replace decision making. From the point 
of view of evaluation theory, this is the reason 
why an evaluation is done in the first place. 

This principle is fully compatible with the 
claim for deliberative democratic evaluation 
by House and Howe (2000), which refers to 
three requirements: the inclusion requirement 
(consider all relevant interests; op. cit., 5f.), the 
dialogue requirement (discover real interests 
through dialogical interaction; op. cit., 6£), and 
the deliberation requirement (values should 
not be taken as given but should be subject to 
examination through rational processes; op. 
cit., p. 8f), and the justification given by the 
two authors can be applied here. 

2. The second principle requires ethical 
values to be respected (domain 1 in the first di¬ 
mension of the taxonomy); some values from 
the social and interactive domain (domain 3) 
might also belong to this field. In case of an 
antinomy within the ethical domain, a prior¬ 
ity must be found and argued for according to 
principle 1. If an ethical value is in antinomy 
with a value from another domain, the evalua¬ 
tor must have strong reasons not to prefer - in 
the case of a gradual antinomy, not to give high 
weight - to the ethical values. If this is not pos¬ 
sible, the evaluator should consider not doing 
the evaluation at all (see Chelimsky, 2006, for 
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svoris priklauso nuo etinio jvertintojo poziu- 
rio ar rimtai apsvarstytij demokratiniij pa- 
stangij (House, Howe, 2000; zr. 1 principq), 
kai tai jmanoma. 

Sis principas paklusta pagrjstq teigi- 
niq principui, pagal kurj vertintojas turi 
atskleisti vertybes, kuriomis vadovaujasi, o 
vertybiij antinomijij atveju reikia pagrjsti 
pasirinktq prioritetq; jis virsesnis uz toliau 
pateiktus principus, nes zmones turi vado- 
vautis etika. 

3. Priemones sugalvojamos vertingu 
laikomam tikslui pasiekti, tad tikslai ture- 
tq buti virsesni uz priemones. Tai atitinka 
I. Kanto antrq kategorinj imperatyvq: .^ki¬ 
tes su kiekviena butybe (tiek su kitais, tiek 
su savimi) taip, kad ji savaime butq tikslas, o 
ne vien tik priemones 1 ' (Kantas, 1785/1987, 
p. 420). Egzistuoja prielaida, kad nera au¬ 
tomating sqsajos tarp metodq ir tikslip; 
panasus metodai gali buti naudojami skir- 
tingiems tikslams, o panasius tikslus gali- 
ma pasiekti skirtingais metodais. Jsitikin^s, 
kad tarp priemoniq ir tikslp yra tiesioginiai 
rysiai, vertintojas turetp tai jrodyti pagal 
1 principp. 

Pavyzdziui, tokius tiesioginius rysius 
tarp priemonip ir tikslp teigia E. Guba ir 
Y. S. Lincoln (2000): vertintojas, pasirink^s 
racionalistinj metateorinj konceptp, - sie- 
jamp su kai kuriais tikslais, - turetp ne- 
naudoti alternatyvaus metodo arba misrip 
metodp pavyzdzio, o vertintojai, pripazjs- 
tantys naturalistinj poziurj, negali naudoti 
palaipsninip ar misrip metodp. Vertinto¬ 
jai gali teikti pranasump skirtingiems da- 
lykams, o taikant sj principp skatinamas 
racionalus dialogas, ar prielaidos yra pa- 
grjstos. Si argumentacija turi didesn^ vert^ 
uz pasirenkamp islygp. 

4. 1-3 laipsnio principal yra formalus, 
o 4-asis nurodo, kokioms konkrecioms ver- 
tybems reiketp suteikti pirmenyb^, tiksliau, 
sietinoms su visuomenes gerove (pvz., Mor¬ 
ris, 2004, p. 367); tai reiskia, kad egzistuojant 


examples). This includes that evaluators must 
not do something that is against their convic¬ 
tion, and they must not be forced to do so. 

This means that the values the stakeholders 
put forward (or would put forward if they could 
be asked; see also principles five and nine) are not 
taken as imperatives, rather they are judged in 
function of weighted values, the weights attrib¬ 
uted to the different values being dependent on 
ethical justification by the evaluator or in a delib¬ 
erative democratic effort (House & Howe, 2000; 
see above, principle 1) as far as this is possible. 

This principle is subordinate to the prin¬ 
ciple of warranted assertibility because the lat¬ 
ter require the evaluator to lay open the val¬ 
ues that lead his or her actions and in case of 
values antinomies to explicitly justify a chosen 
priority; it overrules the other principles below 
because ethics should rule all human action. 

3. Since means are conceived to achieve 
ends which are seen as valuable, the goals 
should be given priority over the means. This 
is in agreement with Kant’s second categorical 
imperative: “Act so that you treat humanity, 
whether in your own person or in that of an¬ 
other, always as an end and never as a means 
only” (Kant, 1785/1987, p. 420 standard pagina¬ 
tion). The premise is that there is no automatic 
link between methods and goals; rather similar 
methods may serve different goals and similar 
goals maybe reached with different methods. If 
an evaluator is convinced there is a one-to-one 
relationship between means and ends, he or she 
should argue for it according to principle 1. 

Such a one-to-one relationship between 
the mean and the ends is assumed, for instance, 
by Guba and Lincoln (2000): An evaluator who 
has chosen a rationalistic meta-theoretical con¬ 
cept - which is associated with some kinds of 
goals - should not use an alternative method or 
a mixed-methods paradigm, nor is it possible 
for naturalistically oriented evaluators to use 
gradual or mixed methods. However, evalua¬ 
tors from the different camps might have differ¬ 
ent preferences, and application of the present 
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vertyhiij antinomijai tarp visuomenes gero¬ 
ves ir kitos vertybes (o aukstesnio laipsnio 
1-3 principal nebesuteikia pirmenybes pas- 
tarajai), vertingesne yra gerove. Vertintojas 
yra atsakingas uz gerov^ tiek kaip pilietis, 
tiekkaip vertintojas (pastarojo negalima at- 
skirti nuo pirmojo). Teigiama, kad stiprinti 
gerov^ - pirmine priezastis, kodel atlieka- 
mas viesiyp program^ vertinimas: gerove 
yra (netiesiogiai arba aiskiai) pagrindine 
daugumos vertinaimj programij vertybe, 
todel vertintojai tureti} elgtis atsakingai. Tai 
nereiskia tiesaus utilitarizmo - kelti zmo- 
nems nepatogumip jeigu bendra geroves 
pusiausvyra teigiama. Labiau siekiama tei- 
singumo, vadovaujantis E. R. House (1976) 
pagal 2 princip^. 

5. Be geroves (4 principas), butina at- 
sizvelgti j visas susijusias vertybes. Tiksliau, 
pabreztina, kad reikia apsvarstyti ne tik so- 
cialinii} dalininkij iskeltas vertybes (jos ais- 
kinamos aptariant 9 princip^). Vertintojas 
privalo nustatyti ir kitas vertybes, kurios 
„statomos ant kortos“, t.y. svarbios; todel 
kaip euristinis fonas gali buti isskirtos sep- 
tynios minetos sritys ir vertybiij pavyzdziai. 
Tuo vertintojas turi patikrinti, ar skirtingu 
sricirj vertybes galima jtraukti j konkretij 
jvertinim^. 

6. Vertybes is teisines srities (4 sritis) pri- 
skiriame tik 6-ajam laipsniui. Esant antino¬ 
mijai srities viduje, atitinkama teisine insti- 
tucija turi issprysti problem^ pries atlikdama 
i vert i ninny Si| sritj priskiriame tik prie 6-ojo 
laipsnio del galimos antinomijos tarp teisines 
vertybes ir vertybes is kitos srities, kuri siame 
s;}rase priskirta aukstesniam laipsniui. Tiks¬ 
liau, jeigu teisine vertybe, grindzianti jver- 
tintLj program^, arba pats jvertinimas pazei- 
dzia visuomenes geroves vertybes, jskaitant 
teisingum;} (pvz., j jcj. jeina zmonii} grupes 
diskriminacija), vertintojas turi apsvarstyti 
galimyby imtis atitinkamij priemoniij, veikti 
kaip pilietis ir jvertinimo proceso dalyvis bei 
prisiimti atsakomyby. 


principle opens the way to a rational argumen¬ 
tation about whether the premises are justified 
or not. Such an argumentation must be pre¬ 
ferred over a seemingly arbitrary stipulation. 

4. While the principles in the ranks 
1 through 3 are formal, the fourth principle 
states which concrete values should have pri¬ 
ority, namely those related with public welfare 
(see e.g. Morris, 2004, 367); this means that if 
there is a values antinomy between public wel¬ 
fare and another value (and the higher ranked 
principle 1 through 3 do not already favor the 
latter), priority must be given to welfare. This 
is because the evaluator has the responsibility 
to promote welfare both as citizen in general 
and as evaluator in particular (which cannot 
be seen independently of the former). It is as¬ 
sumed here that enhancing welfare is the pri¬ 
mary reason why evaluation is performed at 
least when public programs are being evalu¬ 
ated: Welfare is the overriding value, implicitly 
or explicitly, of the most programs to be evalu¬ 
ated, and evaluators should consider not play¬ 
ing the game for programs for which it is not. 
This does not mean a blunt utilitarism in the 
sense that some people might get disadvantag¬ 
es provided that the overall welfare balance is 
positive. Rather, justice in the sense of House 
(1976) is called for according to principle 2. 

5. It is essential to consider in addition to 
welfare (principle 4) all relevant values. In partic¬ 
ular, it must be underlined that not only the val¬ 
ues put forward by stakeholders (which will be 
accounted for in principle 9) need to be consid¬ 
ered but the evaluator must make efforts him- or 
herself to identify further values that might be at 
stake; for this, he or she can use the distinction of 
the seven domains presented above and the ex¬ 
amples of values within them as heuristic back¬ 
ground, the idea being that the evaluator checks 
whether values of the different domains might 
be involved in a particular evaluation. 

6. We put values from the legal domain 
(domain 4) only at the sixth rank. In case of 
an antinomy within the domain, the problem 
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7. Islaidos turi sutapti su biudzetu, 
del kurio susitarta per pradines derybas 
(5 sritis). 

Aisku, kad metodiniai kriterijai nera 
hierarchy os virsuje. 1) „Etiniai klausimai 
(...) turetq tureti pirmenyb^ pries tech- 
ninius jvertinimo atlikimo klausimus" 
(Schwandt, 1997, p. 37); 2) reikia vengti 
metodq, kurie neatsizvelgia j etisko elge- 
sio kriterijus tyrimq ir jvertinimo metu, 
pavyzdziui, kenkia respondentams, mazi- 
na jq savarankiskumq be pagrindo (Patry, 
2002) arba negerbia zmoniq (Morris, 2004, 
p. 367); 3) visada egzistuoja antinomijos 
metodq srities viduje (pvz., jau mineta di- 
apazono apimties tikslumo antinomija, zr. 
ir Patry, 2011), taigi bet kokiu atveju tobu- 
las metodas nera tobulas, o rinktis visada 
reikia. Kita vertus, metodij svarbq pabrezia 
sie argumentai: 4) manome, kad vertinimas 
geriausiai atliekamas per metodinj tinka- 
mumq; metodiniai trukumai gali tureti 
neigiamq pasekmip (zr. ir Simons, 2006). 
Taciau 5) gali prireikti sumazinti metodinj 
kokybtj kitp vertybip naudai. Pavyzdziui, 
atsitiktinai isrinkti kontroliuojami arba 
praktiniai eksperimentai (Campbell, Stan¬ 
ley, 1963), be abejo, yra geriausias budas 
priezastiniams rysiams sukurti (kiek tai 
jmanoma). Kartais tokie projektai tokias si- 
tuacijas, kurios praktiskai buna beprasmes; 
tuomet vertintojas turi pasitenkinti papras- 
tesniu planu (pavyzdziui, kvazieksperi- 
mentiniu), kaip, pavyzdziui, nuolat pabre¬ 
zia M. Scriven (1976), E. R. House (2003) ir 
D. C. Phillips (2006). Labai svarbu priminti 
vertintojui pirmqjj principal renkantis bet 
kokj planq arba kitq metodq, prastesnj nei 
maksimalus sprendimas, reikia tvirtai tai 
pagrjsti - priezastys dazniausiai egzistuoja, 
bet reikia jas aiskiai nurodyti. 

Galiausiai negalima pamirsti asme- 
niniy. socialinip dalininkp vertybii /. Prie 
1-8 principp nurodytos vertybes galio- 
ja bendrai, o asmenines vertybes svarbios 


needs to be solved ahead of the evaluation by 
the appropriate legal institution. It is because 
of the possible antinomy of a legal value with 
a value from another domain which is higher 
ranked in this list that we put this domain only 
at the sixth place. Particularly, if a legal value 
underlying an evaluated program or the evalu¬ 
ation itself violates values of public welfare in¬ 
cluding justice (e.g., contains discrimination 
against groups of people) the evaluator should 
consider taking appropriate measures in his or 
her function and responsibility as citizen and 
as player in the evaluation process. 

7. The costs must fit the budget that was agreed 
upon in the initial negotiation (domain 5). 

8. Methodical criteria are certainly not high¬ 
est in the hierarchy. (1) “(E)thical questions (...) 
should take precedence over technical questions 
of how to do evaluation” (Schwandt, 1997, 37). 
(2) Methods which do not follow the criteria of 
ethical conduct in research and evaluation - for 
example, which harm the respondents, reduce 
their autonomy without justification (see Patry, 
2002) or do not show respect for people (Morris, 
2004, p. 367) - must be avoided. (3) There are 
always antinomies within the methods domain 
(e.g., the bandwidth-fidelity antinomy men¬ 
tioned above, see also Patry, 2011), so the perfect 
method is not possible anyway and choices need 
to be made. The importance of methods, on the 
other hand, is underlined by the following argu¬ 
ments: (4) we suggest that evaluation is best 
served with methodical appropriateness; me¬ 
thodical flaws might have dreadful consequenc¬ 
es (see also Simons, 2006). However, (5), there 
might be the necessity to reduce methodical 
quality in favor of other qualities. For instance, 
randomized controlled or field experiments 
(Campbell & Stanley, 1963) are undoubtedly the 
best way to establish causal relationship (as far 
as this is possible). But sometimes such designs 
engender conditions in the field which do not 
make sense from a practical point of view; un¬ 
der such conditions, the evaluator might settle 
for a less powerful design (for example, quasi- 
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tik socialiniams dalininkams. Bet kuris 
jq, jskaitant vertintojq , j vertinimq butinai 
jtrauks savo vertybes ir interesus: ar aiskiai 
isreikstus, ar numanomus. Is pirmo zvilgs- 
nio sios vertybes atrodo kaip jsitikinimai, 
nors kai kurios jq gali buti gana atspindimos 
ir etiskai jrodinejamos, o kitij socialiniai 
dalininkai gali net neatpazinti, suvokdami 
jas tik jvertinimo proceso metu. Asmenines 
vertybes yra klasifikacijos apacioje, nes kiti 
principal laikomi svarbesniais, bet ne todel, 
kad praktikoje butq nesvarbios. Is tiesq jq 
poveikis labai stiprus ir net gali daryti itakcj. 
kitoms vertybems; taigi per kitus principus 
asmenines vertybes gali jgyti didesnj prio- 
ritetcj.. Socialiniai dalininkai privalo tureti 
savo prioritetus, aiskiai isreiksti ir pagrjsti 
(1 principas), suprasti kitq socialiniq dali- 
ninkq vertybes. Jeigu socialiniai dalininkai 
nepuoseleja vertybiij arba turi tik asmeni- 
nius jsitikinimus, vertintojq pareiga - etis¬ 
kai pagrjsti vertybes ir diskutuojant argu- 
mentuotai aptarti kitvj socialiniij dalininkq 
vertybes (Cronbach ir kt., 1980, p. 171: 
auklejamasis poveikis socialiniams dali¬ 
ninkams; Guba, Lincoln, 1989, p. 56: Ben- 
dras socialiniq dalininkq ugdymas). Pri- 
verstas argumentuoti socialinis dalininkas 
gali buti skatinamas pakeisti vertybes, nors 
tokie pokyciai neturetq lengvai vykti. Ka- 
dangi is sios perspektyvos vertintojas ma- 
tomas ir kaip socialinis dalininkas, jis atlie- 
ka dvejopq vaidmenj: siekia atskleisti visi} 
socialiniij dalininkij ir savo vertybes; kaip 
ekspertas jis gali vaidinti ir treciq vaidmenj, 
pavyzdziui, susijusj su savo institucija. To¬ 
del labai svarbu, kad vertintojas jaustq savo 
vertybes, ypac vertinimo kriterijus, nes sie 
ne visada yra akivaizdus (pig. Speer, 2010, 
p. 415). Tai gali buti svarbu pagal hierarchi- 
jq aukstesniam principui. 

Siais principals reiketq remtis per toles- 
nes diskusijas, galincias pakoreguoti faktq, 
kad „joks jvertinimo teoretikas aiskiai ir sis- 
temingai neatlieka aprasomojo, nurodancio 


experimental), as underlined repeatedly, for 
instance, by Scriven (1976), House (2003) and 
Phillips (2006). It might be particularly impor¬ 
tant to remind the evaluator of the first principle: 
any choice of design or other method below the 
maximal solution according to the state of the 
art requires explicit justification - and in most 
cases there are good reasons for that, but they 
need to be made explicit. 

9. Not to forget, finally, the personal values 
of the stakeholders. While the values involved 
in the principles one through eight are gener¬ 
ally valid, the personal values of the stakeholders 
are claimed only by them. Any stakeholder, in¬ 
cluding the evaluator, will necessarily enter the 
evaluation endeavor with his or her own val¬ 
ues and interests, whether explicit or implicit. 
At first glance, these values are cognitive beliefs, 
although some of them might be quite reflected 
and ethically argued for, while others are not 
even recognized by the stakeholders who might 
become aware of them in the process of evalu¬ 
ation. The reason for ranking these values last 
is because the other principles are seen as more 
important, not because they are unimportant in 
practice; in fact they have a strong influence on 
and may even determine the other values; hence 
they can get a higher priority through other prin¬ 
ciples. The stakeholders should become aware of 
their stakes, they should articulate and justify 
them (principle 1), and they should understand 
the other stakeholders’ values. The evaluator can 
facilitate this process and help the stakeholders 
to evolve from being unaware of their values or 
having just personal beliefs to have ethically jus¬ 
tified values, and their ignorance or the rejection 
of other stakeholders’ values can become a de¬ 
bate based on arguments (Cronbach et al, 1980, 
p. 171: Educative impact on the stakeholders; 
Guba and Lincoln, 1989, p. 56: Mutual education 
of stakeholders). Being forced to argue in such 
cases may also foster a stakeholder to change 
his or her values, although such changes are not 
likely to happen easily. Since in this perspective 
the evaluator is also seen as a stakeholder, he 
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ir metateorinio darbo“ (Shadish ir kt„ 1991, 
p. 49) ir kad etika nera svarbi vadoveliuo- 
se ir straipsniuose apie i vert ini mi) (Simons, 
2006, p. 244). Musi) nuomone, literatura apie 
normatyvines vertybes pernelyg ribota: dau- 
gumoje publikaciji) minima tik nedaugelis 
vertybiij, kiti) nepaisoma arba vertybiq sqra- 
sas pateikiamas be jokio prioriteto (isimtys: 
Morris, 2005; Schwandt, 1997; Simons, 2006; 
ir keletas kitvj). Neneigiame, kad publikaci- 
jos yra vertingos, bet taksonomine struktura 
leidzia vienu metu nagrineti visas vertybes, o 
pateiktieji devyni principal rodo bandymq tai 
padaryti. 

Hierarchija yra dviprasmiska. Mes laiko- 
mes pirmo principo (argumentuotu pagrin- 
dimu), bendrai vadovaujames antru (etikos 
reikalavimais), treciu (tikslij pirmenybe) ir 
ketvirtu (zmoniq geroves) principals bei ma- 
nome, kad butij neatsakinga laikyti juos ze- 
mesniais. Nelaikome to absoliuciu poziuriu, 
nes: a) vis tiek gali buti situaciji), kai pagrjstai 
galima jij nesilaikyti; b) principal suformu- 
luoti labai apibendrintai ir pagal aplinkybes 
jmanoma juos sukonkretinti. Be to, kiekvie- 
nas gali tapti antinomijos dalyku, o vieno 
sprendimo visiems atvejams nera (pavyz- 
dys gali buti diapazono apimties-tikslumo 
antinomija). 

Mes nelaikome sio sqraso baigtiniu; vei- 
kiau tai pirmasis pasiulymas, kurj reikia tobu- 
linti. Apskritai cia pateikti argumentai perne¬ 
lyg trumpi del straipsnio ribotumo. Gali buti, 
kad reikia jtraukti daugiau principij, o kai ku- 
riuos esancius - pasalinti. 


or she has then a double role: to aim at unveil¬ 
ing all stakeholders’ values and to unveil his or 
her own values; as a peer, he or she might have 
a third role, for instance related with his or her 
own institution. In this double or even triple 
role, it is particularly important that the evalua¬ 
tor is aware of his or her values and particularly 
the criteria used in the evaluation given that the 
latter are not always self-evident (cf. Speer, 2010, 
p. 415). This criterion might play important roles 
on a higher principle within this hierarchy. 

These principles are intended as a base for 
further discussion which could remedy to the 
fact that “( n )o evaluation theorist does descrip¬ 
tive, prescriptive, and meta-theorizing explicitly 
and systematically” (Shadish et al„ 1991, p. 49) 
and that ethics are not important in textbooks 
and texts on evaluation (Simons, 2006, p. 244). 
In our view, the literature on normative values is 
much too narrow: most publications deal with a 
small set of values and neglect the other ones, or 
they present just a list of values without priority 
(exceptions are Morris, 2005; Schwandt, 1997; 
Simons, 2006, and some others). This is not 
to deny that such publications are valuable in 
themselves; however, the taxonomic framework 
provides the possibility to consider all these val¬ 
ues simultaneously, and the nine principles pre¬ 
sented above are an attempt to do so. 

The hierarchy is not unambiguous. We stick 
to the principle one (argumentative justifica¬ 
tion) in any case and to the principles two (ethi¬ 
cal requirements), three (priority of the goals), 
and four (human welfare) in general and think 
it would be irresponsible to rank them lower. We 
do not consider this to be a fundamentalist stand¬ 
point because (a) there might still be situations in 
which one can justify to depart from them, and 
(b) they are formulated so broadly that there are 
many ways to concretize them, depending on the 
specific conditions. Further, each of them might 
be subject to antinomies within itself for which 
there is no one single solution for all occasions 
(we have mentioned as example the methods, 
e.g., the bandwidth-fidelity antinomy). 
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We do not regard this list to be final; rather 
it is a first proposition which requires further 
improvement. In any case the argumentation 
here was much too brief due to space con¬ 
straints. It may also be that more principles 
need to be added, and we do not exclude that 
one or the other should be removed at all. 


6 

TAKSONOMIJOS NAUDOJIMAS 
AUKSTOJO MOKSLO INSTITUCIJOMS 
VERTINTI EKSPERTINIU METODU 

Tipiski aukstojo mokslo institucijij eksper- 
tiniai vertinimai susiduria su pernelyg daug 
antinomijq, todel sudetinga nuodugniai jas 
isaiskinti. Toliau bus aptartos prototipine- 
mis laikomos antinomijos, t. y. atspindincios 
problemas, sukuriomis susiduria vertintojai. 
Vadovauj antis dimensijomis, pateiktomis 
2 skyriuje, septynios sritys bus naudojamos 
kaip euristinis pagrindas, identifikavus ati- 
tinkamas vertybes, keturios antinomijos - 
trumpai aptartos. Trecia, vadovaujantis 
hierarchiniais principals, bus aptarta viena 
antinomija. 

6.1. PROTOTIPI NES ANTINOMIJOSTARP 
VERTYBig 

2 skyriuje pristatytos septynios vertybiij sri¬ 
tys; is kiekvienos jij po kelias pateiktos kaip 
pavyzdziai. Galima isskirti maziausiai astuo- 
niasdesimt vertybiij, priesingu vienos kitoms. 
Tai reiskia, kad egzistuoja daugiau nei 3 000 
gal imp kombinacijij, ir dar daugiau, nes kai 
kurios antinomijos apima daugiau nei dvi 
vertybes. Nors ne visos kombinacijos turi 
prasm^, jij gausa rodo, kad klausimai gali 
buti nagrinejami labai kompleksiskai, ban- 
dant antinomijas isanalizuoti visapusiskai. 
Todel jmanoma aptarti tik kelis prototipinius 
pavyzdzius. 


6 

APPLICATION OFTHETAXONOMY 
TO THE EVALUATION OF HIGHER 
EDUCATION INSTITUTIONS 
THROUGH PEER REVIEWING 

In typical peer review evaluations of higher edu¬ 
cation institutions there are too many antinomies 
to give an exhaustive account; hence we discuss 
antinomies which we regard as prototypical, that 
is which represent fairly well the problems that 
evaluators encounter. For this we will proceed 
according to the dimensions presented in sec¬ 
tion 2: The seven domains will serve as heuris¬ 
tic base for identifying relevant values, and four 
antinomies will be discussed briefly. Thirdly, we 
take one of these antinomies and discuss it in 
terms of the hierarchical principles. 

6.1. PROTOTYPICAL ANTINOMIES 
BETWEEN VALUES 

We have presented seven domains of values in 
section 2; each of them has several values, some 
of which have been presented as examples. 
Overall one can distinguish at least eighty val¬ 
ues which can be antagonistic with each other. 
This means that there are more than 3000 pos¬ 
sibilities of combinations, and the number is 
increased since some antinomies address even 
more than two values. Although not all of the 
combinations make sense, the sheer number 
shows the enormous complexity of the issues at 
stake when trying to analyze antinomies com¬ 
prehensively. Therefore it will only be possible 
to discuss some prototypical examples. 
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1PAVYZDYS 

Pirmasis pavyzdys yra is ekonomines srities: 
kiekvienq kart/}, vertinant reikia is anksto nu- 
statyti biudzetq pagal ekspertij darbo kruvi. 
Pries jvertinimq jstaiga pateikia ekspertams 
sutartj, kurioje nurodomi visi isipareigoji- 
mai ir darbai, kuriuos reikia atlikti. jstaigos 
antinomija yra ta, kad siekiama minimi- 
zuoti islaidas, bet pasamdyti pakankamai 
kompetentingij ekspertij ir motyvuoti juos 
dalyvauti velesniuose vertinimuose, jeigu to 
reikes: moketi nei per didelj, nei per mazq 
honorary arba visai jo nemoketi tikintis, 
kad kompetentingi ekspertai gali atsisakyti, 
kadangi tarnauti mokslinei bendruomenei 
visai be atlygio daznai laikoma akademikij 
pareigij dalimi. Antinomija yra palaipsnine 
vienos srities priemoniij dimensijoje. Spren- 
dimas - gauti atlyginimq, pirmiausiai atitin- 
kantj ekspertij atsakomyb^, ypac atsizvelgus 
j asmenin^ dimensijq ir kalbant apie visus 
susijusius asmenis. Tai reiskia, kad honora- 
ras turi buti sijziningas pagal darbo islaidas 
per visij jvertinimo process; tiksli suma pri- 
klauso nuo daug skirtingij dalykij, jskaitant 
pasiulq ir paklausij, nors jos ir nera svar- 
biausios. Vienu konkreciu atveju biudzetas 
buvo lemiama sqlyga, todel ekspertas, j kurj 
buvo kreiptasi (ir kurj jstaiga laike svarbiu), 
atmete pasiulymq. Kadangi atsisakymas ne- 
buvo grindziamas per mazu honoraru (as- 
muo tikriausiai nebutij to pripazin^s), lieka 
neaisku, ar piniginis klausimas isties buvo 
pagrindine atsisakymo nedalyvauti vertini- 
me priezastis. 


2 PAVYZDYS 

Kita antinomija - is teisines srities. Viena 
vertus, jstatymais del duomenq apsaugos 
reikalaujama j ataskaitq itrauktij asmenij 
konfidencialumo, t. y. respondentai pri- 
valo buti neatpazjstami (reikalavimas, at- 
sirad^s, pavyzdziui, ir is etikos principp). 


EXAMPLE 1 

The first example deals with the economic do¬ 
main: for each evaluation a budget has to be 
established in advance, taking into account the 
workload of the reviewers. Ahead of the evalu¬ 
ation the agency proposes a contract with each 
reviewer that includes all obligations and works 
that have to be done. The antinomy of the agency 
is to minimize the expenses yet to engage compe¬ 
tent reviewers in sufficient numbers and to mo¬ 
tivate them for participation in later evaluations 
if necessary: paying the peers not too much and 
not too little honorary - or not to pay them at 
all, with the possible ensuing consequence of the 
refusal of competent peers, although serving the 
scientific community without honorary is often 
regarded as part of the duties of academicians. 
The antinomy is gradual on the means dimension 
within one domain. The solution is to conceive 
a remuneration which is adequate primarily con¬ 
cerning the responsibility the peers have, espe¬ 
cially taking care for the personal dimensions in 
the evaluated institution and regarding all the in¬ 
volved persons; it means the honorary should be 
fair with regards to the operating expenses in the 
complete evaluation process; the exact amount 
depending on many different issues, with offer 
and demand being one of them but not the most 
important; in one particular case the available 
budget was the determining issue, and therefore 
a peer who had been asked (and whom the agen¬ 
cy had regarded as important) declined the offer. 
Since the refusal was not justified with a too low 
honorary (which the person would probably not 
have admitted) it is not clear, however, whether 
the monetary issue indeed was the reason for not 
participating in the evaluation. 

EXAMPLE 2 

The next antinomy is primarily within the legal 
domain. On one hand, the laws on data protec¬ 
tion require confidentiality of the participants 
in the report, i.e., the respondents must not be 
recognizable (this is also a requirement, for 
instance, from ethical principles). However, in 
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Taciau vertinant institucijas, tik keli asme- 
nys uzima tam tikras pareigas, pavyzdziui, 
tera vienas programos direktorius ir pan. 
Kita vertus, pagal vertinimo taisykles (pa- 
prastai neformuluojamas jstatymu), reikia 
atsiskaityti uz svarbius dalykus. Tada atsi- 
randa antinomija: ar pateikti informacijq, 
kuriq zino tik programos direktorius, ar 
j;j atskleisti siekiant isvengti direktoriaus 
identifikavimo? 

Antinomija yra alternatyvi tikslq lygme- 
niu (du tikslai „vertinimo ataskaitos skaityto- 
jas nezino, kas kq atsake" ir „skaitytojas zino, 
kas kq atsake" paneigia vienas kite}, ir neturi 
kompromiso). 

Musi} atveju antinomija buvo isspr^sta 
suteikus informaeijq, taigi ir atskleidus res- 
pondentq. Ekspertas pasiule isvengti anti¬ 
nomy os ir parasyti dvi ataskaitas - vienq su 
visa informaeija (ne anonimin^) institueijai, 
kitg. (anonimin^) - jgaliojusiai jstaigai. Siuo 
konkreciu atveju tai butrj buv£ jmanoma, bet 
antroje ataskaitoje buti} truk§ nemazai esmi- 
nes informaeijos. Tai buvo galima padaryti 
naudojant „sumamj rasymq", siulomq to pa- 
ties eksperto. 

3 PAVYZDYS 

Metodiniu poziuriu lankydamiesi vietoje 
ekspertai nori gauti kuo daugiau pagrjstos 
informaeijos is pokalbiq. Kita vertus, jie turi 
pripazinti interviu davejij tendeneijq neat- 
skleisti informaeijos, galincios jiems pakenkti 
ir sukelti pavojij institueijos egzistavimui; be 
to, galima daryti prielaidq, jog respondentai 
suformuluos informaeijq taip, kad apie juos 
butq susidarytas geras jspudis (Christensen, 
1981). Interviu imantys asmenys turi buti 
mandagus ir neuzduoti „sunkiq“ klausimi} 
(socialine interaktyvi sritis), bet vis delto jie 
privalo klausti tiketinai nemalonii} dalykq 
(metodine sritis). 

Antinomija yra palaipsnine priemo- 
nip lygmeniu tarp sriciij. Ekspertai turi 
buti taktiski (van Manen, 1991) ar jautrus 


evaluations of institutions there are few people 
in certain positions; for instance there is only 
one director of the program etc. On the other 
hand the evaluation rules (usually not formu¬ 
lated as law) require full account of the rele¬ 
vant features. The antinomy is, then, whether 
to give the information that could only be 
available from the director or to disclose it to 
avoid the director being identified. 

The antinomy is alternative on the goal 
level (the two goals “the reader of the evaluation 
report does not know who responded how” 
and “the reader knows who responded how” 
are mutually exclusive, and there is no possi¬ 
bility of a compromise) within one domain. 

In our case, this antinomy was solved by 
giving the information and hence disclosing the 
respondent. A reviewer has suggested as a way 
out of the antinomy to write two reports, one 
with all information (not anonymous) to the 
institution, the other (anonymous) to the man¬ 
dating agency. In this particular case this would 
indeed have been possible; however, for the lat¬ 
ter report substantial information would have 
had to be withheld. This could be done through 
“skillful writing”, as the same reviewer suggests. 

EXAMPLE 3 

From the methodical point of view, during the 
site visit, the peers want as much and as valid 
information in the interview processes as pos¬ 
sible. On the other side they must acknowl¬ 
edge the interviewees’ tendency not to disclose 
information that might be harmful to them 
because it might jeopardize the existence of 
the institution; also one can assume that the 
interviewees will formulate the information 
in a way that it conveys a good impression of 
themselves (Christensen, 1981). The interview¬ 
ers must keep polite and must not ask “hard” 
questions (social interactive domain) yet still 
ask questions that may be disagreeable for the 
interviewees to answer (methodical domain). 

The antinomy is gradual on the means level 
between domains. It requires tact (van Manen, 
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(Forss ir kt., 2002), kad isspnjstq antino¬ 
my surad^ reikiamq pusiausvyrq. Is tiesq 
taktas ir yra pastangos isspr^sti palaipsnin^ 
antinomy (tarp metodiniij ir interaktyviij 
sriciq). Is savo patirties teigiame, kad ne 
visi ekspertai sugebejo tinkamai elgtis to- 
kioje situacijoje. Bet labai sunku patikimai 
ir jtikinamai vertinti, ar ekspertq veiksmai 
yra tinkami, todel cia pateikiame tik asme- 
ninius jspudzius. 

4 PAVYZDYS 

fvertinimo procedures 6 etapo (apsilankyti 
vietoje) metu ekspertas pastebejo, kad stu- 
dentai buvo pasireng^ susitikimui ir j klausi- 
mus atsakinejo nesqziningai, t. y. jq atsakymai 
akivaizdziai priestaravo eksperto zinomiems 
faktams. Kad tinkamai atsakinetij, spaudimq 
jiems galejo daryti institucijos destytojai ir 
vadovas. Kai kurie studentai jautesi nesaugus 
ir pateike dviprasmiskus atsakymus. Vienas 
ekspertas pareikalavo papildomai pasikalbeti 
su vadovu arba destytojais, bet kiti tam ne- 
pritare del riboto laiko ir bijodami neigianuj 
pasekmiiy pavyzdziui, kad bus rizikuojama 
visa jvertinimo procedure, paveikti velesni 
interviu su kitais socialiniais dalininkais ir 
del psichologinio spaudimo, ypac institucijos 
vadovui (jaustis valdomam ir pan.). 

Antinomy tarp dviejq sriciq yra alterna- 
tyvi (etine - tikejimas tuo, kq sako zmones - 
ir metodine) priemoniij lygmeniu. 

Konkreciu atveju ekspertai prieme spren- 
dim;j daugiau nebevertinti ir per ankstes- 
nius pokalbius gautTj informacy pripazinti 
pakankama. 


1991) or “Fingerspitzengefuhl” (Forss et al., 
2002) from the reviewers to solve the antinomy 
by finding the right balance. And tact, indeed, 
is the attempt to deal with a gradual antinomy 
(here between the methodological and the inter¬ 
active domains). In our experience, not all peers 
were able to deal adequately with this situation; 
however, it is very difficult to assess reliably and 
validly whether the peers’ actions are adequate, so 
we refer here only to our personal impression. 

EXAMPLE 4 

During step 6 of the evaluation procedure 
(site visit) one peer got the impression that 
the interviewees were briefed in some kind; 
he noticed that, for instance, the students did 
not answer honestly, i.e., there were obvious 
contradictions between the students’ answers 
and the facts the peers had already acquired. 
There might have been some pressure by the 
teachers and the director of the institution 
on the students to answer in a positive way. 
Some students interacted very insecurely and 
gave ambiguous answers. The peer who was 
particularly sensitive to this issue required an 
additional discussion with the director or the 
teachers. However, the other peers did not ap¬ 
prove such a discussion because of the time 
constraints and because they were afraid of 
negative consequences, such as that the whole 
evaluation procedure might be jeopardized, 
that the further interviews with other stake¬ 
holders might be influenced, and because of 
the psychological pressure especially on the 
director of the institution (feeling under con¬ 
trol etc.). 

The antinomy is alternative between two 
domains (ethic: believing what people say, and 
methodical) on the means level. 

In the particular case the decision of the 
peers was not to proceed to additional as¬ 
sessments but to believe that the information 
gained in the previous interviews with the stu¬ 
dents was sufficient. 
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5 PAVYZDYS 

Kiekvienq sisteminj jvertinimo process 
jskaitant ekspertinj vertinima, sudaro skir- 
tingi etapai. Taigi, pasak H.-J. Abs ir kolegij 
(2006), galutiniai profesionalaus jvertinimo 
pabaigos etapai yra 1) isaiskinti jvertinimq- 
rezultatus ir 2) padaryti isvadas, kad butq 
galima patobulinti vertinimo organizavimq. 
Vertinimo komitetas grjzta j institucijtj. ir 
pateikia isvadas. Institucijos atstovq ir ver- 
tintojq posedyje gali kilti konfliktas tarp as- 
meniniij ir sistemos interesq. Vertintojai turi 
vadovautis sqziningumo ir skaidrumo verty- 
bemis, susitelkti j faktus, surinktus duomenij 
rinkimo etapu, ir vengti insinuaciju. Privalo 
atsizvelgti j tai, kad pristatymas ir konsul- 
tavimas apie pasekmes institucijai savaime 
gali leisti priimti sprendimus del darbuo- 
tojp (teisines vertybes). Taigi sio galutinio 
konsultavimo proceso metu vertintojai turi 
zinoti visij vertinamqjq teiginiij pasekmes 
(Warschburger, 2009). Be to, jq pareiga - 
atsizvelgti j vertinampjp, katedrp vedejp ir 
vadovp rupestj, kitaip sakant, - j asmenines 
jp vertybes. Reikia isaiskinti, kokius vertina¬ 
mpjp jausmus galima uzgauti. Per paskutinj 
ekspertp ir institucijos atstovp susitikimq 
butina islaikyti bendrq abipusio supratimo 
atmosferq. 


6.2. PRINCIPg TAIKYMAS VIENAM 
PAVYZDZIUI 

4 pavyzdziui pritaikyti devyni hierarchiniai 
principal. 

1. Pirmasis principas reikalauja naudoti 
argumentus. Nepriklausomp ekspertp ko- 
manda aptare visus susijusius klausimus. Ta- 
ciau del riboto laiko - gana pavirsutiniskai. 

2. Aptartas etinis poziuris: vertina¬ 
mpjp savarankiskumas (etinis reikalavi- 
mas), o kartu ir studentp instruktavimas, 


EXAMPLE 5 

Each systematic evaluation process including 
peer reviewing follows different steps. So ac¬ 
cording to Abs et al. (2006) the final steps for 
concluding a highly professional evaluation are 

(1) interpretation of the evaluation-results and 

(2) dragging out consequences for improving 
the organization. The evaluation committee re¬ 
turns to the evaluated institution and presents 
the conclusions. Mostly the evaluators and the 
representatives of the institution meet to have 
an analysis session in which the personal inter¬ 
ests and the system interests may conflict. The 
evaluators should follow the value of honesty and 
transparence focusing on facts they gained in the 
data collection phase, and insinuations should be 
avoided. The evaluators have to consider that the 
presentation and the counseling concerning the 
consequences for the institution itself might yield 
decisions about the personnel (legal values). So 
the evaluators have to be aware of the conse¬ 
quences of all the evaluative statements in this 
final counseling process (Warschburger, 2009). It 
is also the evaluators’ duty to consider the anxi¬ 
ety of the evaluees, the heads of departments and 
the director as well; this means considering then- 
personal values. It has to be clarified what might 
be at stake with respect to the feelings of the eval¬ 
uees. A general atmosphere of mutual under¬ 
standing is essential within the different feedback 
loops of the final meeting between the peers and 
the representatives of the institution. 

6.2. APPLICATION OF THE PRINCIPLES 
TO ONE EXAMPLE 

The nine hierarchic principles will be applied, 
again for demonstrative purposes, to Example 4. 

1. The first principle requires the use of 
arguments. Within the peer review team, all 
relevant issues were put on the table and dis¬ 
cussed. However, time constraints restricted 
the depth of the discussion. 

2. Next, the ethical view needs to be dis¬ 
cussed: The autonomy of the evaluees (an ethical 
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netinkamas etiniu poziuriu. Vis delto, 
tam ekspertai neturejo jtakos. Galiausiai 
pasitikejimas zmonemis irgi vertinamas 
kaip etikos principas. Preliminariai del 
etiniq klausimq pirmenybes (ir del ribo- 
to laiko) buvo nuspr^sta toliau problemos 
nenagrineti. 

3. Su atitinkamomis priemonemis buvo 
siejami du tikslai: vienu atzvilgiu - pasitike¬ 
jimas ir santykiai, kitu - metodinis pagrjstu- 
mas. Taigi treciasis principas nepadejo, nes 
antinomija buvo „tikslas pries tikslcj". 

4. Ketvirtasis principas - visuomenes 
gerove - rizikingas, nes neigiamas ekspertij 
sprendimas turetq neigiamq pasekmiq ins- 
titucijai, taigi ir susijusiij asmenij gerovei 
(destytojams ir studentams). Iskeltas klausi- 
mas - priimti ar atmesti institucijos akredi- 
tavimq ir teiginiij pagrjstumq (kuriems buvo 
kil^s pavojus sioje antinomijoje) - tam tikra 
prasme buvo sprendziamasis. Sprendimas 
neakredituoti galetq pakenkti asmenij, ku- 
riuos tiesiogiai liecia sprendimas, socialinei 
gerovei, bet jis gali ir tureti naudos visuome- 
nei, jeigu del jo institucijoje daromi radika- 
lus pokyciai j gera. Siuo atveju atsizvelgta ir 
j kitp informacijq, todel visuomenes geroves 
klausimas nebuvo laikomas esminiu siai 
konkreciai antinomijai; vis delto visuome¬ 
nes gerove laikyta svarbia priimant galutinj 
sprendimq. 

5. Kalbant apie visas susijusias vertybes: 
socialiniai dalininkai is institucijos norejo, 
kad vyktq kuo maziau pokyciq. Valstybe, 
mokanti uz institucijp, pageidavo optimalios 
institucijos, del to kai kurie pakeitimai gale- 
jo buti butini (pagal vertintojp nuomon^), 
ir pusiausvyros tarp investicijp ir rezultatp. 
Vertintojai norejo surinkti pagrjstus duome- 
nis nesugais^ pernelyg daug laiko. Institucijos 
nariai nenorejo nei skirti daug laiko, nei jokip 
pokycip. Toks principas nepadeda issprpsti 
antinomijos. 

6. Aptariant teisinius klausimus nebuvo 
jokip nesklandump. 


requirement) was put forward, but also the prob - 
lem of briefing the students which is inappropri¬ 
ate from the ethical point of view - the reviewers, 
however, had no influence on that. Finally, trust¬ 
ing people was also seen as an ethical principle. 
Tentatively, because of the priority of the ethical 
issues (but also because of time constraints), not 
following up the problem was decided. 

3. Two ends were related with the respec¬ 
tive means: trust and relationship on one hand 
and methodical validity on the other. The third 
principle, hence, did not help because the an¬ 
tinomy was “aim against aim”. 

4. Public welfare, the fourth principle, was 
at stake insofar as a wrong decision of the peers 
would have negative consequences for the insti¬ 
tution and hence for the welfare of the people 
involved (teachers as well as students). The 
question was whether to accept or refuse ac¬ 
creditation of the institution and the validity of 
the statements (which was at stake in this an¬ 
tinomy) was decisive for this to some degree. 
Deciding to refuse accreditation might harm 
the social welfare of those directly touched by 
the decision, but it might also benefit society 
if the decision leads to a radical change of the 
institution towards the better. In the particular 
case, other information was also taken into ac¬ 
count so that the issue of public welfare was not 
considered crucial for this particular antinomy; 
however, public welfare was considered impor¬ 
tant when making the final decision. 

5. Considering all relevant values: The 
stakeholders from the institution wanted to 
keep the institution with as little change as 
possible. The state who pays for the institution 
wanted an optimal institution, which may or 
may not make some changes necessary (de¬ 
pending on the evaluators’ judgments), and a 
good investment-outcome balance. The evalu¬ 
ators wanted valid data, but not to invest too 
much time. The institution members, them¬ 
selves, did not want to invest much time and 
did not want any change. The principle does 
not help to solve the antinomy. 
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7. Vertinimui skirtas laikas („vertinimo 
istekliai") - stiprus argumentas, kad nereikia 
daugiau vertinimij, bet sutarta, jog lankyma- 
sis vietoje - pernelyg trumpas. 

8. Zvelgiant is metodinio tasko, - butij 
reikej^ daugiau vertinimi}, taciau kiti mineti 
suvarzymai laikyti svarbesniais. 

Aptartuose principuose buvo atsizvelgta j 
visas asmenines socialiniy. dalininki} vertybes. 

Baigiant ir aptariant visas vertybes pa- 
gal principq hierarchy nuspr^sta: geriau 
neatlikti tradicinii} vertinimij. Vis delto 
nutarta atsizvelgti j skirtingus teiginius ir 
mandagiai paaiskinti (pig., pedagoginj tak- 
tq), kad negalima visiskai pasikliauti infor- 
macija. Taigi vadovui uzsiminta, jog paste- 
betos pastangos instruktuoti studentus ir tai 
laikyta netinkamu elgesiu, taciau nesukurta 
neigiama atmosfera tarp institucijos, nepri- 
klausomij ekspertij komandos ir jgaliojan- 
cios jstaigos. 

Principij hierarchijos taikymas vienai 
antinomijai pirmiausia rodo, kad kai kurie 
principai gali buti nereiksmingi, o kitus ap- 
tarti - gana sudetinga, todel reikia atsizvelg¬ 
ti j papildomq informacija, pateiktq prie 
4 principo (visuomenes gerove). Apskritai 
principq hierarchija (mazesnes numeracijos 
principij prioritetas pries didesnes numera¬ 
cijos principus) buvo naudinga. Bet reikia 
pamineti, kad tik vienas vertintojas zinojo 
principij hierarchijq priimant sprendimus 
ir diskutuodamas su kitais ekspertais ja 
nesinaudojo. 


6. In the discussion legal issues were not at 
stake. 

7. The time allotted for the evaluation (“re¬ 
sources for evaluation”) was a strong argument 
against doing more assessments, yet it was 
agreed that the length of the site visit was too 
short. 

8. From the methodical point of view more 
assessments would have been necessary. Yet 
the other constraints mentioned above were 
seen as more important. 

9. All personal values by the stakeholders 
were taken into account in the principles dis¬ 
cussed above. 

In conclusion, and considering all values 
according to the hierarchy of principles, it 
was decided that it was better not to have ad¬ 
ditional assessments. Nevertheless, it was also 
decided to take into account, in the report, the 
relative validity of the different statements and 
to make clear, in a polite way (cf. pedagogical 
tact), that the information was not completely 
trusted. Thus the director was given the hint 
that the attempt to brief the students had been 
discovered and that this was seen as inappro¬ 
priate, without creating a negative atmosphere 
between the institution, the peer review team, 
and the mandating agency. 

The application of the hierarchy of prin¬ 
ciples to one antinomy shows, first, that some 
principles can be irrelevant while the discus¬ 
sion of others can become quite complicated, 
requiring taking into account additional infor¬ 
mation, as seen in principle 4 (public welfare). 
Overall, the hierarchy of the principles (prior¬ 
ity of the lower-numbered principles over the 
higher-numbered ones) was helpful. It must be 
mentioned here, however, that only one of the 
evaluators was aware of the hierarchy of prin¬ 
ciples at the time of decision making and that 
this person did not use the principles in the 
argumentation with the other peers. 
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DISKUSIJA 

Aptarta keturiij dimensijy antinomijy tak- 
sonomija: vertyhiij sritys, alternatyvios ir 
palaipsnines antinomijos, antinomijos tiksly 
ir priemoniij lygmenimis bei principij hie¬ 
rarchy a, turinti padeti nustatyti prioritetus, 
susidurus su antinomy a. Sios taksonomijos 
naudojimas iliustruotas penkiais pavyzdziais, 
kuriy vienas isnagrinetas detaliau. Is pavyz- 
dziy matyti, kad dimensijos yra tarpusavyje 
susijusios. Tiksliau, tarp sriciy (pirma dimen¬ 
sija) ir principy (ketvirta dimensija) yra dali- 
nis sutapimas. Tad kokia prasme atskirti abi 
dimensijas? 

Pirmiausia reikia pabrezti, kad sioje 
taksonomijoje pirmoji dimensija nera nu- 
rodanti. Ji veikiau turi buti vertinama kaip 
kontrolinis syrasas, norint uztikrinti, jog bus 
atsizvelgta j visas vertybes, galincias tureti 
tam tikros jtakos. Literaturine ir ekspertiniy 
vertinimij patirtis rodo, kad vertintojai lin- 
apsiriboti keliomis vertybemis ir nepri- 
pazjsta viso vertybiy, jtraukty j vertinimy, 
spektro. Norint racionaliai argumentuoti, 
reikia atsizvelgti j visas vertybes, kurios gali 
buti susijusios. Tai tarsi kolektyvinio naujy 
idejy svarstymo procedura (Osborne, 1957): 
pirmiausia elemental identifikuojami, ve- 
liau - vertinami. Vertinimo metu, atsizvel- 
giant j vertybes, reikia pereiti tris etapus: 1) 
vertybiy identifikavimy (1 dimensija); 2) ga- 
limy antinomijy tarp siy vertybiy identifi¬ 
kavimy ir antinomijy tipy nustatymy (2 ir 3 
dimensijos); ir 3) sprendimy, kaip elgtis su 
atitinkamomis antinomy omis, vadovauj an¬ 
tis principals (4 dimensija). Sie etapai rodo, 
kad dimensijos savaime nera nepriklauso- 
mos, nors jy naudojimas jvertinimo procese 
ir skiriasi. 

Kokia prasme kurti tokiy daugiamat^ 
hierarchin^ taksonomijy? Pirmiausia noretu- 
me pabrezti, kas ji nera: 

• vertinimo instrumentas. Kaip buvo 
mineta, skirtingos klasifikacijos yra gana 


DISCUSSION 

A taxonomy of antinomies consisting in four 
dimensions was presented: domains of values; 
alternative and gradual antinomies; antino¬ 
mies on the goal and on the means levels; and 
a hierarchy of principles that should help to set 
priorities when confronted with an antinomy. 
The use of this taxonomy has been illustrated 
with five examples, one of which has been 
analyzed more in detail. As these latter ex¬ 
amples show, the dimensions are interrelated. 
In particular, there is an overlap between the 
domains (first dimension) and the principles 
(fourth dimension). What sense does it make, 
then, to distinguish these two dimensions? 

First of all, it must be underlined that in 
this taxonomy the first dimension is not pre¬ 
scriptive, but rather must be seen as a checklist 
to ensure that all values that might play a role 
are taken into account. Experience both from 
the literature as well as from peer review evalu¬ 
ations show that evaluators tend to focus on few 
values and do not recognize the full scope of 
values that are involved in an evaluation. To ar¬ 
gue rationally about values requires to take into 
account all those which might be relevant. It is 
like in a brain-storming procedure (Osborne, 
1957): First, the elements are identified; then 
they are judged. In evaluations, with respect 
to values, three steps are necessary: (1) Identi¬ 
fying the values (dimension 1); (2) identifying 
possible antinomies between these values and 
identifying the types of antinomies (dimensions 
2 and 3); and (3) deciding how to deal with the 
respective antinomies based on the principles 
(dimension 4). These steps show that while the 
dimensions per se are not independent, their 
use in an evaluation process is quite different. 

What sense does it make to formulate such 
a multi-dimensional hierarchic taxonomy? 
First we want to stress what it is not: 

• It is not an assessment instrument. As 
mentioned repeatedly above, the different 
classifications are not as unambiguous as would 
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vienprasmes. Musij nuomone, is sios struktu- 
ros jmanoma sukurti vertinimo instrument^, 
taciau tam reiketq daug aiskesniq kategorijij, 
ypac kalbant apie sritis. Bet patirtis rodo, kad 
nuomones skiriasi, ar antinomija yra alterna- 
tyvi, ar palaipsnine, ar ji tikslo, ar priemoniu 
lygmens. Vadinasi, vienas ekspertas tq pack} 
antinomija gali laikyti alternatyvia, kitas - 
palaipsnine. Pavyzdziui, E. Guba ir Y. S. Lin¬ 
coln (2000) teigia, kad naturalistinis pavyzdys 
neapima racionalistinio, kitvj nuomone, jma¬ 
noma integruoti bent kelias is pagrindinii} 
„aksiomi}“ (del vertinimo zr.: Shadish ir kt., 
199,1 p. 46-52; delholizmo zr.: Phillips, 2000, 
3 sky rip; del ideografijos zr.: Patry, Gastager, 
2002); 

• tinkama metavertinimams atlikti, 
nei lyginti, nei vertybems vertinti del ver¬ 
tinimo modelio ar pastangq teigiant, kad 
tam tikras principas labiau tinka X nei Y 
jvertinti, arba kritikuojant, kad nesilaiko- 
ma hierarchijos. Tai netinkamas vertini- 
mas, nes nebutq atsizvelgta j visq situacijos 
kompleksiskumq. 

• taisyklii} rinkinys, kuriuo aklai turi 
sekti jvertintojai. Tiksliau, optimalus palaips- 
ninirj antinomija sprendimas gali skirtis pa- 
gal situacijLj (Patry, 1992); 

• kriterijij rinkinys, padedantis nusprys- 
ti, kurj vertinimo pasiulymq rinktis. Takso- 
nomija - ne klausimu rinkinys, kuriuos reikia 
isnagrineti teikiant vertinimo pasiulym;j arba 
ataskaitq; 

• tai, kuo butq ketinama is sociali- 
niq dalininkr}, ypac vertintojq, atsisakyti 
atsakomybes. 

Naudotis siomis funkcijomis be pateisini- 
mo laikoma piktnaudziavimu. Nors nurodyti 
tikslai gali buti teiseti - ne musij atsakomybe 
tai nuspr^sti, - taksonomija nera tinkama to- 
kiam tikslui. 

Tad kam ji gali buti naudojama? Pirma, 
norime paskatinti diskusijtj. apie vertinto- 
ji} bendruomenes vertybes. Diskusija jau 
vyksta, bet, musq manymu, per vangiai, nes 


be necessary for this. It would be possible, we 
think, to develop an assessment instrument from 
this framework but this would require much 
clearer categories, particularly with respect to 
the domains; experience shows, however, that 
people also disagree whether an antinomy is al¬ 
ternative or gradual and whether it is on the goal 
or on the means dimension. This means that the 
same antinomy may be alternative for one judge 
but gradual for another one; for instance, Guba 
and Lincoln (2000) claim that their naturalistic 
paradigm excludes the rationalistic one while 
others believe that an integration is possible at 
least for some of their key “axioms” (for valuing 
see Shadish et aL, 1991 pp. 46-52; regarding hol¬ 
ism, see Phillips, 2000, chapter 3; for idiography, 
see Patry and Gastager, 2002). 

• For this reason it would also be inap¬ 
propriate to do meta-evaluations using this 
taxonomy, neither for comparative purposes 
nor to make values judgments about a given 
evaluation model or endeavor, by saying such 
and such principle is applied to a higher de¬ 
gree in evaluation X than in evaluation Y or by 
criticizing that the hierarchy is not respected. 
Such a judgment would be inappropriate be¬ 
cause it would not take into account the whole 
complexity of the situation. 

• It is not a set of rules that must be fol¬ 
lowed blindly by evaluators. Particularly the 
optimum solution in gradual antinomies may 
differ from situation to situation (Patry, 1992). 

• It is not a set of criteria to decide which 
evaluation proposal to prefer in an open sub¬ 
scription, nor is it a set of issues that must be 
addressed in any case in an evaluation pro¬ 
posal or in an evaluation report. 

• And finally, and most importantly, it is 
not intended to take away the responsibilities 
of any of the stakeholders including, particu¬ 
larly, the evaluators. 

Any use in these functions without fur¬ 
ther justification should be considered as 
abuse. While the goals expressed there might 
be legitimate - we do not want to judge 
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publikacijq autoriai susitelkia tik j specifi- 
nius, juos dominancius klausimus. Nors sios 
diskusijos yra svarbios, siulome padiskutuoti 
ir apie vertybes visose vertybiq srityse. Be to, 
musij pabrezti skirtumai (dimensija ir an- 
tinomijos tipas) gali suteikti diskusijai tiks- 
lumo: kai kuriems debatams truksta tiksliq 
argument^, pavyzdziui, analize atskleistq, 
kad pazeistos elementarios mokslines kalbos 
taisykles. Must} patirtis rodo, kad ekspertq 
jsitikinimai daznai buna klaidingi, del to jie 
eina „gincytinais trumpiausiais keliais“, t. y. 
pernelyg greitai padaro isvadas, todel to- 
kie argumentai jautrus kritikai. Naudojama 
pagrjsta metaetine struktura prideda argu- 
mentui tikslumo (zr. 1 principal garantuotas 
tvirtinimas). 

Antra, taksonomijos funkcija - suteik¬ 
ti vertintojij demesj j vertybes, j kurias jis 
paprastai neatsizvelgtq. Kai kuriuos daly- 
kus galima ignoruoti, bet siai mokslinei 
veiklai - vertinimui - budinga pagrjsti pa- 
sirinkim;} argumentais, todel taksonomija 
gali buti naudojama kaip orientacija, kq dar 
galima paaiskinti. Siuo atzvilgiu ji laikoma 
euristine priemone, kuri pritaria nuomonei, 
jog vertybes turi buti aptartos per vertini- 
mo process arba skirtingiems poziuriams 
atstovaujanciq vertintojq (pvz., is skirtingu 
„jvertinimo medzio sakq“, Alkin, Christie, 
2004). 

Bet svarbiausia, kad taksonomija gali buti 
naudojama kaip bandomasis kontrolinis scj- 
rasas del specifinio praktinio vertybii} ir anti- 
nom ijij jvertinimo ir kaip pagalba stengiantis 
jas isnagrineti. Del sios priezasties pateikiame 
tarn tikros srities - aukstojo mokslo instituci- 
jp ekspertinio vertinimo - pavyzdzkj. Kaip 
mineta, si sritis pernelyg sudetinga, kad ver¬ 
tybii} antinomijos buti} issamiai paaiskintos. 
Net aptariant vieni} ivertinimt}, antinomiji} 
(nepriklausomai, ar jos buti} akivaizdzios ar 
aiskiai nurodytos), netiesiogines (santykiai 
nera aiskus) arba slaptos (dalyviai nezino apie 
antinomijas; pig. Miller, 2006, p. 20, del sio 


about this, - he taxonomy by itself is just not 
appropriate for such a purpose. 

What can it be used for, then? First, we want 
to stimulate the discussion about values within 
the evaluator community. This discussion already 
exists, but we estimate it to be much too narrow, 
each publication or author focusing only on spe¬ 
cific issues which are of special interest to him or 
her. We suggest that, in addition to such discus¬ 
sions which are important, comprehensive dis¬ 
cussions of values issues across all values domains 
be performed. We suggest also that the use of the 
distinctions that were made (dimension and type 
of antinomy) can contribute to precision in the 
discussion. Indeed, at least some of the debates 
lack in precise argumentation; an analysis would 
reveal, for instance, that elementary rules of scien¬ 
tific language are violated. It is our experience that 
peers often commit the naturalistic fallacy which 
leads them to use “argumentative shortcuts”, i.e., 
to jump too quickly to conclusions; this makes the 
argumentation string sensitive to criticism. Using 
an appropriate meta-ethical framework would 
add to the strength of their argument (see prin¬ 
ciple 1: warranted assertibility). 

Second, the taxonomy has the function to 
focus the attention of evaluators on values issues 
that he or she might have not taken into account 
otherwise. Some issues might have been disre¬ 
garded for good reasons in a particular evalua¬ 
tion process; it is appropriate for a scientific en¬ 
deavor like evaluation, however, that choices are 
justified through arguments, so the taxonomy 
can serve as orientation what possibly should 
also be accounted for. In this sense it is thought of 
as being a heuristic tool to support values discus¬ 
sions within an evaluation process or between 
evaluators, for example, representing different 
approaches (e.g., from different “branches” in 
Alkin and Christies, 2004, “evaluation tree”). 

Most importantly, however, the taxonomy 
can be used as a tentative checklist for values 
and antinomies in specific practical evaluations 
and as a help when trying to deal with these an¬ 
tinomies. For this reason we have attempted to 
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isskyrimo) butij pernelyg daug, kad galetu- 
me isnagrineti siame straipsnyje. Tad sutelk^ 
demesj ne j antinomijij turinj, o j process, 
teigiame, jog taksonomija siuo atzvilgiu yra 
vertinga. 

Principal siulomi kaip pagalba sprendi- 
mams del vertybiij antinomijij priimti. Kitaip 
tariant, manome, jog jvertinusi diskusijas 
bendruomene pritars patobulintam principij 
rinkiniui pagal argumentus ir pagrindim^. 
Tada vertintojas galetij ne tik naudotis prin¬ 
cipals kaip pagalba, priimdamas sprendimus, 
bet ir isdestyti argumentus, kuriij, tikriausiai, 
kitaip butij nepaisoma, t. y. antinomineje situ- 
acijoje taksonomija primintij ne tik vertybes, 
bet ir argumentus pries arba uz jas. Veliau rei- 
ketij aiskiai argumentuoti pries arba uz tam 
tikras vertybes tik tuo atveju, jei jos nukryptij 
nuo sip principij. 

Straipsnyje pateikta taksonomija siuloma 
kaip priemone, galinti padeti praktiskai pri¬ 
imti sprendinuj del vertybiij vertinimo. Sios 
priemones labai reikia vertinimo bendruo- 
menei, zinoma, jeigu vertintojai nori pripa- 
zinti vertybiij antinomijas, su kuriomis susi- 
duria dirbdami. 
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provide examples from a particular field, peer 
reviews of higher education institutions. As ar¬ 
gued above, the field is much too complex to pro¬ 
vide an exhaustive account of the values antino¬ 
mies; even if we had taken one single evaluation 
the number of antinomies - whether manifest 
(stated explicitly), indirect (the relations are not 
made explicit) or latent (the participants are not 
aware of the antinomies; cf. Miller, 2006, p. 20, 
for this distinction) - would outnumber by far 
what can be dealt with in such a paper. But the 
focus is not on the content of the antinomies, 
rather it is on the process, and we assume that 
the taxonomy is valuable in this regard. 

The principles are suggested as help in 
making decisions in values antinomies. In 
particular, we imagine that after some discus¬ 
sion the evaluation community will agree on 
an improved set of principles with according 
arguments and justifications. Then an evalua¬ 
tor would not only have the principles them¬ 
selves as help in decision-making, but he or she 
would also dispose of arguments possibly ne¬ 
glected otherwise, that is, the taxonomy would 
not only be a reminder of values but also of ar¬ 
guments for or against them in an antinomic 
situation. Further, it would be necessary to ar¬ 
gue explicitly in favor or against certain values 
only if he or she departs from these principles. 

Overall, hence, the taxonomy presented 
here is proposed as a tool that should help prac¬ 
tical decision-making for values issues in evalu¬ 
ations. Such a tool is desperately needed in the 
evaluation community, provided that the evalu¬ 
ators are willing to face the values antinomies 
they encounter when working in the field. 
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